[Libvir] [RFC][PATCH 1/2] NUMA memory and topology patches

[PATCH 1/2] - add capability to access free memory information on each NUMA cell. Signed-off-by: Beth Kon (eak@us.ibm.com) -- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com

Hi, I have one question. Why you include xend_internal.h in xen_internal.c? Is this not avoidable from your logic? Thanks Atsushi SAKAI beth kon <eak@us.ibm.com> wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
Signed-off-by: Beth Kon (eak@us.ibm.com)
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com

Atsushi SAKAI wrote:
Hi,
I have one question. Why you include xend_internal.h in xen_internal.c? Is this not avoidable from your logic?
You're right that as the code stands now it is not needed. Originally I had the call to xenDaemonNodeGetInfo in xenHypervisorNodeGetCellsFreeMemory, so it was needed. Richard Jones is suggesting that it be moved back there, so once we decide on that, I will remove the include if appropriate.
Atsushi SAKAI
beth kon <eak@us.ibm.com> wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
Signed-off-by: Beth Kon (eak@us.ibm.com)
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com

beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
+ +/* number of cells in the node will be saved for later use */ +int nbNodeCells; This should be static I think. + if (xenDaemonNodeGetInfo(conn, &nodeInfo)) { + /* BETH - not sure what would be the appropriate error type. should I add one for xend? */ + virXendError(conn, VIR_ERR_XEN_CALL, " cannot determine actual number of cells"); + return -1; + } It's certainly possible to add errors (to virterror) if none of the existing errors seem to fit. But in this case I think it looks OK. A bigger problem here is that I think the call to xenDaemonNodeGetInfo should happen inside xen_internal.c:xenHypervisorNodeGetCellsFreeMemory, something like this: xenHypervisorNodeGetCellsFreeMemory (virConnectPtr conn, ....) { static int nb_nodes = -1; if (nb_nodes == -1) { virNodeInfo info; if (xenDaemonNodeGetInfo (conn, &info) == -1) { ... (error) ... } nb_nodes = info->nodes; } ... } Note also the explicit test for == -1 The rest of the patch looks OK to me. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903

Richard W.M. Jones wrote:
beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
+ +/* number of cells in the node will be saved for later use */ +int nbNodeCells;
This should be static I think.
As the code stands now, I don't think so. It is defined in xend_internal.c but used in xen_internal.c
A bigger problem here is that I think the call to xenDaemonNodeGetInfo should happen inside xen_internal.c:xenHypervisorNodeGetCellsFreeMemory, something like this:
xenHypervisorNodeGetCellsFreeMemory (virConnectPtr conn, ....) { static int nb_nodes = -1;
if (nb_nodes == -1) { virNodeInfo info; if (xenDaemonNodeGetInfo (conn, &info) == -1) { ... (error) ... } nb_nodes = info->nodes; }
... }
Note also the explicit test for == -1
I like this solution. I thought that Daniel wanted it out of the xenHypervisorNodeGetCellsFreeMemory call, and in some init code. Daniel? Do you have an opinion on this?
The rest of the patch looks OK to me.
Rich.
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com

On Tue, Sep 25, 2007 at 09:20:37AM -0400, beth kon wrote:
Richard W.M. Jones wrote:
beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
+ +/* number of cells in the node will be saved for later use */ +int nbNodeCells;
This should be static I think.
As the code stands now, I don't think so. It is defined in xend_internal.c but used in xen_internal.c
A bigger problem here is that I think the call to xenDaemonNodeGetInfo should happen inside xen_internal.c:xenHypervisorNodeGetCellsFreeMemory, something like this:
xenHypervisorNodeGetCellsFreeMemory (virConnectPtr conn, ....) { static int nb_nodes = -1;
if (nb_nodes == -1) { virNodeInfo info; if (xenDaemonNodeGetInfo (conn, &info) == -1) { ... (error) ... } nb_nodes = info->nodes; }
... }
Note also the explicit test for == -1
I like this solution. I thought that Daniel wanted it out of the xenHypervisorNodeGetCellsFreeMemory call, and in some init code. Daniel? Do you have an opinion on this?
Not really. The number of cells on the host is currently only useful to only one routine, so either solution is fine by me. In general I try to avoid exporting global variables, in that case a simple routine set in xen_unified.c and exported from xen_unified.h could do that lookup and keep the value as a static variable. That would allow to share as in your code, while avoiding a global variable like in Rich suggestion. But at this point this is nitpicking a bit, we will have to refactor this slightly anyway I think :-) Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

On Mon, Sep 24, 2007 at 11:29:07PM -0400, beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
Signed-off-by: Beth Kon (eak@us.ibm.com)
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com
diff -urpN libvirt.danielpatch/include/libvirt/libvirt.h.in libvirt.cellsMemory/include/libvirt/libvirt.h.in --- libvirt.danielpatch/include/libvirt/libvirt.h.in 2007-09-11 15:29:43.000000000 -0400 +++ libvirt.cellsMemory/include/libvirt/libvirt.h.in 2007-09-20 14:11:32.000000000 -0400 @@ -586,8 +586,9 @@ int virDomainDetachDevice(virDomainPtr d */
int virNodeGetCellsFreeMemory(virConnectPtr conn, - unsigned long *freeMems, - int nbCells); + long long *freeMems, + int startCell, + int maxCells);
Hum, I just realize something, shouldn't we use unsigned long long *freeMems instead. I don't see what semantic a negative value can carry, but maybe I missed something ! Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Daniel Veillard wrote:
On Mon, Sep 24, 2007 at 11:29:07PM -0400, beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
Signed-off-by: Beth Kon (eak@us.ibm.com)
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com
diff -urpN libvirt.danielpatch/include/libvirt/libvirt.h.in libvirt.cellsMemory/include/libvirt/libvirt.h.in --- libvirt.danielpatch/include/libvirt/libvirt.h.in 2007-09-11 15:29:43.000000000 -0400 +++ libvirt.cellsMemory/include/libvirt/libvirt.h.in 2007-09-20 14:11:32.000000000 -0400 @@ -586,8 +586,9 @@ int virDomainDetachDevice(virDomainPtr d */
int virNodeGetCellsFreeMemory(virConnectPtr conn, - unsigned long *freeMems, - int nbCells); + long long *freeMems, + int startCell, + int maxCells);
Hum, I just realize something, shouldn't we use unsigned long long *freeMems instead. I don't see what semantic a negative value can carry, but maybe I missed something !
Daniel
Yes, I agree. I'll change that. -- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com

On Mon, Sep 24, 2007 at 11:29:07PM -0400, beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell. [...] diff -urpN libvirt.danielpatch/src/xen_internal.c libvirt.cellsMemory/src/xen_internal.c --- libvirt.danielpatch/src/xen_internal.c 2007-09-11 15:29:44.000000000 -0400 +++ libvirt.cellsMemory/src/xen_internal.c 2007-09-24 22:04:05.000000000 -0400 [...] * Returns the number of entries filled in freeMems, or -1 in case of error. */ int -xenHypervisorNodeGetCellsFreeMemory(virConnectPtr conn, unsigned long *freeMems, - int nbCells) +xenHypervisorNodeGetCellsFreeMemory(virConnectPtr conn, long long *freeMems, + int startCell, int maxCells) { - if ((conn == NULL) || (freeMems == NULL) || (nbCells < 0)) - return -1; + xen_op_v2_sys op_sys; + int i, j, ret; + xenUnifiedPrivatePtr priv; +
+ if ((conn == NULL) || (maxCells < 1) || (startCell > nbNodeCells)) { + virXenErrorFunc (VIR_ERR_INVALID_ARG, __FUNCTION__, + "invalid argument", 0); + return -1; + } /* - * TODO: - * - get the number of cell in the node - * - if not NUMA returns the available memeoy directly in freeMems[0] - * return 1 - * - if NUMA iterates over the cells using a specific hypercall - * filling up entries until full or at the end of the NUMA cells + * Support only sys_interface_version >=4 */ - return(-1); + if (sys_interface_version < 4) { + virXenErrorFunc (VIR_ERR_NO_XEN, __FUNCTION__, + "unsupported in sys interface < 4", 0); + return -1; + }
I found out that using VIR_ERR_NO_XEN here isn't a good idea, first it's not that Xen hypervisoris not found but that this specific call is not available, and second virsh filter out automatically VIR_ERR_NO_XEN. It is better to use VIR_ERR_XEN_CALL , you then get: [root@test2 src]# ./virsh freecell 1 libvir: Xen error : failed Xen syscall xenHypervisorNodeGetCellsFreeMemory: unsupported in sys interface < 4 0 [root@test2 src]# echo $? 1 [root@test2 src]# Which is a correct error report IMHO. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Daniel Veillard wrote:
On Mon, Sep 24, 2007 at 11:29:07PM -0400, beth kon wrote:
[PATCH 1/2] - add capability to access free memory information on each NUMA cell.
[...]
diff -urpN libvirt.danielpatch/src/xen_internal.c libvirt.cellsMemory/src/xen_internal.c --- libvirt.danielpatch/src/xen_internal.c 2007-09-11 15:29:44.000000000 -0400 +++ libvirt.cellsMemory/src/xen_internal.c 2007-09-24 22:04:05.000000000 -0400
[...]
* Returns the number of entries filled in freeMems, or -1 in case of error. */ int -xenHypervisorNodeGetCellsFreeMemory(virConnectPtr conn, unsigned long *freeMems, - int nbCells) +xenHypervisorNodeGetCellsFreeMemory(virConnectPtr conn, long long *freeMems, + int startCell, int maxCells) { - if ((conn == NULL) || (freeMems == NULL) || (nbCells < 0)) - return -1; + xen_op_v2_sys op_sys; + int i, j, ret; + xenUnifiedPrivatePtr priv; +
+ if ((conn == NULL) || (maxCells < 1) || (startCell > nbNodeCells)) { + virXenErrorFunc (VIR_ERR_INVALID_ARG, __FUNCTION__, + "invalid argument", 0); + return -1; + } /* - * TODO: - * - get the number of cell in the node - * - if not NUMA returns the available memeoy directly in freeMems[0] - * return 1 - * - if NUMA iterates over the cells using a specific hypercall - * filling up entries until full or at the end of the NUMA cells + * Support only sys_interface_version >=4 */ - return(-1); + if (sys_interface_version < 4) { + virXenErrorFunc (VIR_ERR_NO_XEN, __FUNCTION__, + "unsupported in sys interface < 4", 0); + return -1; + }
I found out that using VIR_ERR_NO_XEN here isn't a good idea, first it's not that Xen hypervisoris not found but that this specific call is not available, and second virsh filter out automatically VIR_ERR_NO_XEN. It is better to use VIR_ERR_XEN_CALL , you then get:
[root@test2 src]# ./virsh freecell 1 libvir: Xen error : failed Xen syscall xenHypervisorNodeGetCellsFreeMemory: unsupported in sys interface < 4 0
[root@test2 src]# echo $? 1 [root@test2 src]#
Which is a correct error report IMHO.
Daniel
Ok, that sounds good. -- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com
participants (4)
-
Atsushi SAKAI
-
beth kon
-
Daniel Veillard
-
Richard W.M. Jones