[libvirt] [PATCH] fix size parameter in calls to numa_node_to_cpus
I was getting the error: map size mismatch; abort when running the daemon-conf tests. I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source). The attached patch fixes the problem. Dave diff --git a/src/qemu_conf.c b/src/qemu_conf.c index e6c378f..7019c14 100644 --- a/src/qemu_conf.c +++ b/src/qemu_conf.c @@ -324,7 +324,7 @@ qemudCapsInitNUMA(virCapsPtr caps) for (n = 0 ; n <= numa_max_node() ; n++) { - if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0) goto cleanup; for (ncpus = 0, i = 0 ; i < MAX_CPUS ; i++) diff --git a/src/uml_conf.c b/src/uml_conf.c index 7eb630d..3659c6b 100644 --- a/src/uml_conf.c +++ b/src/uml_conf.c @@ -80,7 +80,7 @@ umlCapsInitNUMA(virCapsPtr caps) for (n = 0 ; n <= numa_max_node() ; n++) { - if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0) goto cleanup; for (ncpus = 0, i = 0 ; i < MAX_CPUS ; i++)
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ? I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities) Applying your patch and it still works correctly which is good, so ACK to applying it from that point of view. I'd still like to understand why I'm not seeing the error message, but you are... Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
"Daniel P. Berrange" <berrange@redhat.com> wrote:
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ?
I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities)
Applying your patch and it still works correctly which is good, so ACK to applying it from that point of view. I'd still like to understand why I'm not seeing the error message, but you are...
Same question here. At first I just assumed it was because I hadn't been using --with-numactl (I installed numactl-devel on that system just yesterday), but even now, I can't reproduce the failure. This is on rawhide (though same version on F10): $ rpm -q numactl-devel numactl-devel-2.0.2-2.fc10.x86_64
Daniel P. Berrange wrote:
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ?
I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities)
Applying your patch and it still works correctly which is good, so ACK to applying it from that point of view. I'd still like to understand why I'm not seeing the error message, but you are...
Daniel
I'm puzzled by why I'm the only one seeing it as well. One difference may be that I seem to have compiled --with-uml. I'm not sure if that's the way most people are doing things; it's just what I get from autobuild.sh on my box. I'm still tracking down an unresolved issue on my setup during the daemon-conf scripts, which is that libvirtd is getting farther into running during the corrupt configuration tests than Jim thinks it should. I'll take a look to see if that is the reason for this behavior as well. Dave
On Thu, Dec 11, 2008 at 03:07:11PM -0500, Dave Allan wrote:
Daniel P. Berrange wrote:
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ?
I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities)
Applying your patch and it still works correctly which is good, so ACK to applying it from that point of view. I'd still like to understand why I'm not seeing the error message, but you are...
Daniel
I'm puzzled by why I'm the only one seeing it as well. One difference may be that I seem to have compiled --with-uml. I'm not sure if that's the way most people are doing things; it's just what I get from autobuild.sh on my box.
UML actually defaults to on, so I have that too. In fact all drivers now default to on, provided their compile pre-requisites exist. Are you x86_64 too ? Are you on an actual NUMA enabled machine ? My machines are NUMA enabled.
I'm still tracking down an unresolved issue on my setup during the daemon-conf scripts, which is that libvirtd is getting farther into running during the corrupt configuration tests than Jim thinks it should. I'll take a look to see if that is the reason for this behavior as well.
FYI, I know there is at least 2 crash conditions in the daemon currently due to my threading changes. I've got patches almost ready for those... Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Dave Allan <dallan@redhat.com> wrote:
Daniel P. Berrange wrote:
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ?
I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities)
Applying your patch and it still works correctly which is good, so ACK to applying it from that point of view. I'd still like to understand why I'm not seeing the error message, but you are...
Daniel
I'm puzzled by why I'm the only one seeing it as well. One difference may be that I seem to have compiled --with-uml. I'm not sure if that's the way most people are doing things; it's just what I get from autobuild.sh on my box.
I've just confirmed that when configure runs here, it enables UML support (I see that it's the default, and there is no prerequisite): $ grep UML..= config.status D["WITH_UML"]=" 1" and yet, all the tests pass for me.
On Thu, Dec 11, 2008 at 07:43:01PM +0000, Daniel P. Berrange wrote:
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ?
I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities)
I saw the problem too on one machine, it broke make check , coincidentally it s also an F-10 x86_64 :-) 138) xml2sexprdata/xml2sexpr-fv-sound.xml OK Validated 138 files, 0 failed PASS: domainschematest 1) fc4.conf ... OK 2) libvirtd.conf ... OK PASS: test_conf.sh PASS: cpuset /u/veillard/libvirt/tests/daemon-conf-30173 testing with corrupted config: listen_tls --- expected-err 2008-12-12 11:30:30.000000000 +0100 +++ err 2008-12-12 11:30:30.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in1.conf: listen_tls: invalid type: got string; expected long testing with corrupted config: listen_tcp --- expected-err 2008-12-12 11:30:30.000000000 +0100 +++ err 2008-12-12 11:30:30.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in2.conf: listen_tcp: invalid type: got string; expected long testing with corrupted config: tls_port --- expected-err 2008-12-12 11:30:30.000000000 +0100 +++ err 2008-12-12 11:30:30.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in3.conf: tls_port: invalid type: got long; expected string testing with corrupted config: tcp_port etc .... and with the patch everything works again on that machine (Intel core duo with 2GB memory). Still I hope we were not allocating a gigabyte of memory for that call (if yes there is a serious fix needed even with a factor of 8 less). So applied and commited, thanks a lot, but I'm still puzzled, why did we end out of memory, I will look quickly Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On Fri, Dec 12, 2008 at 11:40:23AM +0100, Daniel Veillard wrote:
On Thu, Dec 11, 2008 at 07:43:01PM +0000, Daniel P. Berrange wrote:
On Thu, Dec 11, 2008 at 01:56:47PM -0500, Dave Allan wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
I'm still very puzzelled by this. What OS and architecture are you running on ?
I'm using Fedora 10, x86_64 and don't see any messages with current CVS and it returns NUMA info correctlys (as per virsh capabilities)
I saw the problem too on one machine, it broke make check , coincidentally it s also an F-10 x86_64 :-)
138) xml2sexprdata/xml2sexpr-fv-sound.xml OK Validated 138 files, 0 failed PASS: domainschematest 1) fc4.conf ... OK 2) libvirtd.conf ... OK PASS: test_conf.sh PASS: cpuset /u/veillard/libvirt/tests/daemon-conf-30173 testing with corrupted config: listen_tls --- expected-err 2008-12-12 11:30:30.000000000 +0100 +++ err 2008-12-12 11:30:30.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in1.conf: listen_tls: invalid type: got string; expected long
testing with corrupted config: listen_tcp --- expected-err 2008-12-12 11:30:30.000000000 +0100 +++ err 2008-12-12 11:30:30.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in2.conf: listen_tcp: invalid type: got string; expected long
testing with corrupted config: tls_port --- expected-err 2008-12-12 11:30:30.000000000 +0100 +++ err 2008-12-12 11:30:30.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in3.conf: tls_port: invalid type: got long; expected string
testing with corrupted config: tcp_port
etc ....
and with the patch everything works again on that machine (Intel core duo with 2GB memory). Still I hope we were not allocating a gigabyte of memory for that call (if yes there is a serious fix needed even with a factor of 8 less).
So applied and commited, thanks a lot, but I'm still puzzled, why did we end out of memory, I will look quickly
The 'out of memory' message is a red-herring - its a bogus error reporting path after the failure. Something that also needs fixing... Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Dec 12, 2008 at 10:46:27AM +0000, Daniel P. Berrange wrote:
On Fri, Dec 12, 2008 at 11:40:23AM +0100, Daniel Veillard wrote:
So applied and commited, thanks a lot, but I'm still puzzled, why did we end out of memory, I will look quickly
The 'out of memory' message is a red-herring - its a bogus error reporting path after the failure. Something that also needs fixing...
numa_node_to_cpus in the new version of libnuma seems to be a macro doing more checks than the older entry point, umlCapsInitNUMA() returns -1 and we assumed it was possible only if running out of memory. I guess reporting a normal error is better, note the patch will touch the strerror cleanup, Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
Dave Allan <dallan@redhat.com> wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
The attached patch fixes the problem.
Dave diff --git a/src/qemu_conf.c b/src/qemu_conf.c index e6c378f..7019c14 100644 --- a/src/qemu_conf.c +++ b/src/qemu_conf.c @@ -324,7 +324,7 @@ qemudCapsInitNUMA(virCapsPtr caps)
for (n = 0 ; n <= numa_max_node() ; n++) {
- if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0)
Hi Dave, [Sorry I didn't catch this earlier. ] With that patch, I am now seeing that out of memory message in "make check" test failures, e.g., testing with corrupted config: min_workers --- expected-err 2008-12-12 15:30:20.000000000 +0100 +++ err 2008-12-12 15:30:20.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in23.conf: min_workers: invalid type: got string; expected long Since we're using NUMA_VERSION1_COMPATIBILITY mode (anyone know why?) #if HAVE_NUMACTL #define NUMA_VERSION1_COMPATIBILITY 1 #include <numa.h> #endif numa_node_to_cpus expands to a call to numa_node_to_cpus_compat, which is defined in numa.h: static inline int numa_node_to_cpus_compat(int node, unsigned long *buffer, int buffer_len) { struct bitmask tmp; tmp.maskp = (unsigned long *)buffer; tmp.size = buffer_len * 8; return numa_node_to_cpus(node, &tmp); } So the question is, what units to use in "tmp.size". To answer that, I looked above, also in numa.h at this: struct bitmask { unsigned long size; /* number of bits in the map */ unsigned long *maskp; }; And since MAX_CPUS_MASK_LEN is a buffer length, in bytes, from qemu_conf.c: qemudCapsInitNUMA(virCapsPtr caps) { ... if (VIR_ALLOC_N(mask, MAX_CPUS_MASK_LEN) < 0) goto cleanup; and tmp.size is measured in bits, we have to continue to use that MAX_CPUS_MASK_LEN parameter, and not divide it by 8: So I propose this change: (with it, "make check" does pass once again)
From ceae19496cd60bd9a7b3527046b42e9f8460ae82 Mon Sep 17 00:00:00 2001 From: Jim Meyering <meyering@redhat.com> Date: Fri, 12 Dec 2008 16:22:34 +0100 Subject: [PATCH] Revert "fix bits/bytes memory request mismatches"
* src/qemu_conf.c (qemudCapsInitNUMA): Do pass MAX_CPUS_MASK_LEN as buffer length in call to numa_node_to_cpus. * src/uml_conf.c (umlCapsInitNUMA): Likewise. --- src/qemu_conf.c | 2 +- src/uml_conf.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/qemu_conf.c b/src/qemu_conf.c index 59171e7..206fb0b 100644 --- a/src/qemu_conf.c +++ b/src/qemu_conf.c @@ -324,7 +324,7 @@ qemudCapsInitNUMA(virCapsPtr caps) for (n = 0 ; n <= numa_max_node() ; n++) { - if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) goto cleanup; for (ncpus = 0, i = 0 ; i < MAX_CPUS ; i++) diff --git a/src/uml_conf.c b/src/uml_conf.c index 3659c6b..7eb630d 100644 --- a/src/uml_conf.c +++ b/src/uml_conf.c @@ -80,7 +80,7 @@ umlCapsInitNUMA(virCapsPtr caps) for (n = 0 ; n <= numa_max_node() ; n++) { - if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) goto cleanup; for (ncpus = 0, i = 0 ; i < MAX_CPUS ; i++) -- 1.6.0.4.1044.g77718
On Fri, Dec 12, 2008 at 04:29:43PM +0100, Jim Meyering wrote:
Dave Allan <dallan@redhat.com> wrote:
Since we're using NUMA_VERSION1_COMPATIBILITY mode (anyone know why?)
#if HAVE_NUMACTL #define NUMA_VERSION1_COMPATIBILITY 1 #include <numa.h> #endif
In Fedora 10, they introduced libnuma 2.x, whereas all previous Fedora had 1.x. libnuma 2.x changed API decl for various functions. libvirt is written against 1.x API, so we need to declare NUMA_VERSION1_COMPATIBILITY to make sure our header import on 2.x systems, uses the 1.x APIs. The black magic in the header, actually compiles our 1.x usage into calls to the 2.x APIs. The 1.x ABIs still existed as versioned symbols so if you compiled against 1.x, you can still run against 2.x. Fun isn't it :-) Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Jim Meyering wrote:
Dave Allan <dallan@redhat.com> wrote:
I was getting the error:
map size mismatch; abort
when running the daemon-conf tests.
I traced it to two places in which we were passing a parameter as a number of bits to numa_node_to_cpus which expects a number of bytes (see numa_node_to_cpus_compat in numa.h in the numactl source).
The attached patch fixes the problem.
Dave diff --git a/src/qemu_conf.c b/src/qemu_conf.c index e6c378f..7019c14 100644 --- a/src/qemu_conf.c +++ b/src/qemu_conf.c @@ -324,7 +324,7 @@ qemudCapsInitNUMA(virCapsPtr caps)
for (n = 0 ; n <= numa_max_node() ; n++) {
- if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0)
Hi Dave,
[Sorry I didn't catch this earlier. ]
With that patch, I am now seeing that out of memory message in "make check" test failures, e.g.,
testing with corrupted config: min_workers --- expected-err 2008-12-12 15:30:20.000000000 +0100 +++ err 2008-12-12 15:30:20.000000000 +0100 @@ -1,2 +1,5 @@ +map size mismatch; abort +: Success +umlStartup: out of memory remoteReadConfigFile: in23.conf: min_workers: invalid type: got string; expected long
Since we're using NUMA_VERSION1_COMPATIBILITY mode (anyone know why?)
#if HAVE_NUMACTL #define NUMA_VERSION1_COMPATIBILITY 1 #include <numa.h> #endif
numa_node_to_cpus expands to a call to numa_node_to_cpus_compat, which is defined in numa.h:
static inline int numa_node_to_cpus_compat(int node, unsigned long *buffer, int buffer_len) { struct bitmask tmp;
tmp.maskp = (unsigned long *)buffer; tmp.size = buffer_len * 8; return numa_node_to_cpus(node, &tmp); }
So the question is, what units to use in "tmp.size". To answer that, I looked above, also in numa.h at this:
struct bitmask { unsigned long size; /* number of bits in the map */ unsigned long *maskp; };
And since MAX_CPUS_MASK_LEN is a buffer length, in bytes,
from qemu_conf.c: qemudCapsInitNUMA(virCapsPtr caps) { ... if (VIR_ALLOC_N(mask, MAX_CPUS_MASK_LEN) < 0) goto cleanup;
and tmp.size is measured in bits, we have to continue to use that MAX_CPUS_MASK_LEN parameter, and not divide it by 8:
So I propose this change: (with it, "make check" does pass once again)
From ceae19496cd60bd9a7b3527046b42e9f8460ae82 Mon Sep 17 00:00:00 2001 From: Jim Meyering <meyering@redhat.com> Date: Fri, 12 Dec 2008 16:22:34 +0100 Subject: [PATCH] Revert "fix bits/bytes memory request mismatches"
* src/qemu_conf.c (qemudCapsInitNUMA): Do pass MAX_CPUS_MASK_LEN as buffer length in call to numa_node_to_cpus. * src/uml_conf.c (umlCapsInitNUMA): Likewise. --- src/qemu_conf.c | 2 +- src/uml_conf.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/qemu_conf.c b/src/qemu_conf.c index 59171e7..206fb0b 100644 --- a/src/qemu_conf.c +++ b/src/qemu_conf.c @@ -324,7 +324,7 @@ qemudCapsInitNUMA(virCapsPtr caps)
for (n = 0 ; n <= numa_max_node() ; n++) {
- if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) goto cleanup;
for (ncpus = 0, i = 0 ; i < MAX_CPUS ; i++) diff --git a/src/uml_conf.c b/src/uml_conf.c index 3659c6b..7eb630d 100644 --- a/src/uml_conf.c +++ b/src/uml_conf.c @@ -80,7 +80,7 @@ umlCapsInitNUMA(virCapsPtr caps)
for (n = 0 ; n <= numa_max_node() ; n++) {
- if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN / 8) < 0) + if (numa_node_to_cpus(n, mask, MAX_CPUS_MASK_LEN) < 0) goto cleanup;
for (ncpus = 0, i = 0 ; i < MAX_CPUS ; i++) -- 1.6.0.4.1044.g77718
With the patch reverted, I am back to having the errors. I'll track it down. I must have a different numa version from you guys. Dave
participants (4)
-
Daniel P. Berrange -
Daniel Veillard -
Dave Allan -
Jim Meyering