[libvirt] virsh nodeinfo CPU detection questions
I have a machine with four AMD Opteron(TM) 6272 processors: http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P... I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4: 0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]# 1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]# As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this? Numa cell(s) X CPU Socket(s) == physical CPUs present However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this? Core(s) per socket X Thread(s) per core: == CPU cores Thanks in advance for clarifying.
On 01/29/13 16:52, Scott Sullivan wrote:
I have a machine with four AMD Opteron(TM) 6272 processors:
http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P...
I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4:
0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]#
This version probably didn't detect the NUMA nodes correctly.
1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]#
As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this?
Numa cell(s) X CPU Socket(s) == physical CPUs present
To detect the total physical count of processors (packages) you can't rely on this output. You should refer to the recently added topology information in the capabilities XML (accessible through the command "virsh capabilities"). The NUMA topology section there holds information (specially "socket_id") that allow you to count the number of physical sockets on the machine. Unfortunately, this output will be part of tomorrows release :)
However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this?
To determine the core count, you also need to refer to the information in capabilities. Unfortunately AMD processors (especially Bulldozer ones cannot be detected reliably with the code as it is now).
Core(s) per socket X Thread(s) per core: == CPU cores
Thanks in advance for clarifying.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Jan 29, 2013 at 9:52 AM, Scott Sullivan <ssullivan@liquidweb.com> wrote:
I have a machine with four AMD Opteron(TM) 6272 processors:
http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P...
I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4:
0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]#
1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]#
As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this?
Numa cell(s) X CPU Socket(s) == physical CPUs present
However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this?
Core(s) per socket X Thread(s) per core: == CPU cores
Thanks in advance for clarifying.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Scott, I've got the exact same CPUs (Dell PowerEdge R815). You'll find the NUMA info is detected node info is detected incorrectly. Oddly we have different data detected. What kernel are you running? with libvirt 1.0.1 root@host ~ # virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2100 MHz CPU socket(s): 1 Core(s) per socket: 64 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 132013200 KiB virsh capabilities is a bit "smarter" but still not 100%. It sees 8 NUMA nodes and puts the CPUs in the correct nodes but still detects the CPU incorrectly. -- Doug Goldstein
On 01/29/2013 02:55 PM, Doug Goldstein wrote:
On Tue, Jan 29, 2013 at 9:52 AM, Scott Sullivan<ssullivan@liquidweb.com> wrote:
I have a machine with four AMD Opteron(TM) 6272 processors:
http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P...
I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4:
0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]#
1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]#
As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this?
Numa cell(s) X CPU Socket(s) == physical CPUs present
However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this?
Core(s) per socket X Thread(s) per core: == CPU cores
Thanks in advance for clarifying.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Scott,
I've got the exact same CPUs (Dell PowerEdge R815). You'll find the NUMA info is detected node info is detected incorrectly. Oddly we have different data detected. What kernel are you running?
with libvirt 1.0.1 root@host ~ # virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2100 MHz CPU socket(s): 1 Core(s) per socket: 64 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 132013200 KiB
virsh capabilities is a bit "smarter" but still not 100%. It sees 8 NUMA nodes and puts the CPUs in the correct nodes but still detects the CPU incorrectly.
Doug, I'm running a RedHat kernel; 2.6.32-220.2.1.el6.x86_64 specifically. Thanks for the insight of your experiences, very interesting. I find the same that the NUMA info appears detected incorrectly for this processor.
On Wed, Jan 30, 2013 at 12:53 PM, Scott Sullivan <ssullivan@liquidweb.com> wrote:
On 01/29/2013 02:55 PM, Doug Goldstein wrote:
On Tue, Jan 29, 2013 at 9:52 AM, Scott Sullivan<ssullivan@liquidweb.com> wrote:
I have a machine with four AMD Opteron(TM) 6272 processors:
http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P...
I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4:
0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]#
1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]#
As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this?
Numa cell(s) X CPU Socket(s) == physical CPUs present
However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this?
Core(s) per socket X Thread(s) per core: == CPU cores
Thanks in advance for clarifying.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Scott,
I've got the exact same CPUs (Dell PowerEdge R815). You'll find the NUMA info is detected node info is detected incorrectly. Oddly we have different data detected. What kernel are you running?
with libvirt 1.0.1 root@host ~ # virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2100 MHz CPU socket(s): 1 Core(s) per socket: 64 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 132013200 KiB
virsh capabilities is a bit "smarter" but still not 100%. It sees 8 NUMA nodes and puts the CPUs in the correct nodes but still detects the CPU incorrectly.
Doug,
I'm running a RedHat kernel; 2.6.32-220.2.1.el6.x86_64 specifically.
Thanks for the insight of your experiences, very interesting. I find the same that the NUMA info appears detected incorrectly for this processor.
Scott, It appears that the CPU info isn't detected correctly because it can't be detected correctly or really expressed in the values we know/have. For some info you can see: https://bugzilla.redhat.com/show_bug.cgi?id=874050 The issue is that libvirt and to extent the Linux kernel expect 1 physical socket to equate to 1 NUMA node. In the case of these processors there are really 2 individual "packages" in 1 physical socket. Each package has its own memory controller and that is tied to 8 cores. Hence how you get 8 NUMA nodes with 8 processors each. However to add to the confusion AMD has implemented their own type of hyperthreading in that they have two separate cores but 2 cores share 1 FPU. So in reality each package has 8 cores but if you're doing FPU work, you only have 4 per package. The previous decision was that 4 sockets, 8 cores per socket and 2 thread per core was not a correct way to express this CPU since it is true that each core separate. But the current solution is IMHO worse by saying 1 socket, 64 cores, 1 thread per core. A better result would be to say 8 sockets, 8 cores per socket, and 1 thread per core. But that would require us to break the idea that a socket is a physical socket and align a socket with a "package". Anyone else thoughts? Would this be acceptable? Right now the output is broken and if you put any NUMA values into the XML or use numad the domain will not start and will give a cryptic error message. Which is a lot worst than reporting 4 sockets, 8 cores, 1 thread or 8 sockets, 8 cores and 1 thread IMHO. -- Doug Goldstein
On Wed, Jan 30, 2013 at 4:12 PM, Doug Goldstein <cardoe@gentoo.org> wrote:
On Wed, Jan 30, 2013 at 12:53 PM, Scott Sullivan <ssullivan@liquidweb.com> wrote:
On 01/29/2013 02:55 PM, Doug Goldstein wrote:
On Tue, Jan 29, 2013 at 9:52 AM, Scott Sullivan<ssullivan@liquidweb.com> wrote:
I have a machine with four AMD Opteron(TM) 6272 processors:
http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P...
I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4:
0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]#
1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]#
As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this?
Numa cell(s) X CPU Socket(s) == physical CPUs present
However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this?
Core(s) per socket X Thread(s) per core: == CPU cores
Thanks in advance for clarifying.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Scott,
I've got the exact same CPUs (Dell PowerEdge R815). You'll find the NUMA info is detected node info is detected incorrectly. Oddly we have different data detected. What kernel are you running?
with libvirt 1.0.1 root@host ~ # virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2100 MHz CPU socket(s): 1 Core(s) per socket: 64 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 132013200 KiB
virsh capabilities is a bit "smarter" but still not 100%. It sees 8 NUMA nodes and puts the CPUs in the correct nodes but still detects the CPU incorrectly.
Doug,
I'm running a RedHat kernel; 2.6.32-220.2.1.el6.x86_64 specifically.
Thanks for the insight of your experiences, very interesting. I find the same that the NUMA info appears detected incorrectly for this processor.
Scott,
It appears that the CPU info isn't detected correctly because it can't be detected correctly or really expressed in the values we know/have. For some info you can see: https://bugzilla.redhat.com/show_bug.cgi?id=874050
The issue is that libvirt and to extent the Linux kernel expect 1 physical socket to equate to 1 NUMA node. In the case of these processors there are really 2 individual "packages" in 1 physical socket. Each package has its own memory controller and that is tied to 8 cores. Hence how you get 8 NUMA nodes with 8 processors each. However to add to the confusion AMD has implemented their own type of hyperthreading in that they have two separate cores but 2 cores share 1 FPU. So in reality each package has 8 cores but if you're doing FPU work, you only have 4 per package.
The previous decision was that 4 sockets, 8 cores per socket and 2 thread per core was not a correct way to express this CPU since it is true that each core separate. But the current solution is IMHO worse by saying 1 socket, 64 cores, 1 thread per core. A better result would be to say 8 sockets, 8 cores per socket, and 1 thread per core. But that would require us to break the idea that a socket is a physical socket and align a socket with a "package".
Anyone else thoughts? Would this be acceptable? Right now the output is broken and if you put any NUMA values into the XML or use numad the domain will not start and will give a cryptic error message. Which is a lot worst than reporting 4 sockets, 8 cores, 1 thread or 8 sockets, 8 cores and 1 thread IMHO.
-- Doug Goldstein
Its worth noting that even with the new capabilities data in 1.0.2, this is still behaving incorrectly when you attempt to start the VM and have ANY numa configuration in the XML for the domain. kvm_init_vcpu failed: Cannot allocate memory -- Doug Goldstein
On 01/30/13 23:12, Doug Goldstein wrote:
On Wed, Jan 30, 2013 at 12:53 PM, Scott Sullivan <ssullivan@liquidweb.com> wrote:
On 01/29/2013 02:55 PM, Doug Goldstein wrote:
On Tue, Jan 29, 2013 at 9:52 AM, Scott Sullivan<ssullivan@liquidweb.com> wrote:
I have a machine with four AMD Opteron(TM) 6272 processors:
http://www.amd.com/us/products/server/processors/6000-series-platform/6200/P...
I use 'virsh nodeinfo' to determine the amount of physical CPUs, and CPU cores. What values should I be looking at in the output of nodeinfo to determine this? With 1.X libvirts the output values have changed from 0.9.4:
0.9.4: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 4 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 1 Memory size: 24713980 kB [root@host28 ~]#
1.0.1: [root@host ~]# virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2099 MHz CPU socket(s): 1 Core(s) per socket: 8 Thread(s) per core: 2 NUMA cell(s): 4 Memory size: 24713980 KiB [root@host28 ~]#
As you can see, the CPU socket and NUMA cells values have changed in these two versions, on the same machine. To get the physical CPU count, is it safe to assume this?
Numa cell(s) X CPU Socket(s) == physical CPUs present
However, I am not positive what formula I should use to determine the CPU core count. On this processor, it has 16 cores. Would that mean I can do this?
Core(s) per socket X Thread(s) per core: == CPU cores
Thanks in advance for clarifying.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Scott,
I've got the exact same CPUs (Dell PowerEdge R815). You'll find the NUMA info is detected node info is detected incorrectly. Oddly we have different data detected. What kernel are you running?
with libvirt 1.0.1 root@host ~ # virsh nodeinfo CPU model: x86_64 CPU(s): 64 CPU frequency: 2100 MHz CPU socket(s): 1 Core(s) per socket: 64 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 132013200 KiB
virsh capabilities is a bit "smarter" but still not 100%. It sees 8 NUMA nodes and puts the CPUs in the correct nodes but still detects the CPU incorrectly.
Doug,
I'm running a RedHat kernel; 2.6.32-220.2.1.el6.x86_64 specifically.
Thanks for the insight of your experiences, very interesting. I find the same that the NUMA info appears detected incorrectly for this processor.
Scott,
It appears that the CPU info isn't detected correctly because it can't be detected correctly or really expressed in the values we know/have. For some info you can see: https://bugzilla.redhat.com/show_bug.cgi?id=874050
The issue is that libvirt and to extent the Linux kernel expect 1 physical socket to equate to 1 NUMA node. In the case of these processors there are really 2 individual "packages" in 1 physical socket. Each package has its own memory controller and that is tied to 8 cores. Hence how you get 8 NUMA nodes with 8 processors each. However to add to the confusion AMD has implemented their own type of hyperthreading in that they have two separate cores but 2 cores share 1 FPU. So in reality each package has 8 cores but if you're doing FPU work, you only have 4 per package.
The previous decision was that 4 sockets, 8 cores per socket and 2 thread per core was not a correct way to express this CPU since it is true that each core separate. But the current solution is IMHO worse by saying 1 socket, 64 cores, 1 thread per core. A better result would be to say 8 sockets, 8 cores per socket, and 1 thread per core. But that would require us to break the idea that a socket is a physical socket and align a socket with a "package".
The data in the "nodeinfo" structure (retrirevable by "virsh nodeinfo") have to add up to the total number of processors in the host if you multiply all the fields. This is due to historic reasons.
Anyone else thoughts? Would this be acceptable? Right now the output is broken and if you put any NUMA values into the XML or use numad the domain will not start and will give a cryptic error message. Which is a lot worst than reporting 4 sockets, 8 cores, 1 thread or 8 sockets, 8 cores and 1 thread IMHO.
To get correct data, you have to use libvirt-1.0.2 with this script: https://bugzilla.redhat.com/attachment.cgi?id=690965&action=edit
On 02/05/2013 04:34 AM, Peter Krempa wrote:
The data in the "nodeinfo" structure (retrirevable by "virsh nodeinfo") have to add up to the total number of processors in the host if you multiply all the fields. This is due to historic reasons.
Hmm - we document under virNodeInfo the cases where the return value will not be ideal, as well as how to detect that issue: struct _virNodeInfo { char model[32]; /* string indicating the CPU model */ unsigned long memory; /* memory size in kilobytes */ unsigned int cpus; /* the number of active CPUs */ unsigned int mhz; /* expected CPU frequency */ unsigned int nodes; /* the number of NUMA cell, 1 for unusual NUMA topologies or uniform memory access; check capabilities XML for the actual NUMA topology */ unsigned int sockets; /* number of CPU sockets per node if nodes > 1, 1 in case of unusual NUMA topology */ unsigned int cores; /* number of cores per socket, total number of processors in case of unusual NUMA topology*/ unsigned int threads; /* number of threads per core, 1 in case of unusual numa topology */ }; but the documentation for virNodeGetInfo could probably use a reminder sentence (so that the problem is obvious just from the function documentation, without having to chase through links to the struct documentation).
Anyone else thoughts? Would this be acceptable? Right now the output is broken and if you put any NUMA values into the XML or use numad the domain will not start and will give a cryptic error message. Which is a lot worst than reporting 4 sockets, 8 cores, 1 thread or 8 sockets, 8 cores and 1 thread IMHO.
To get correct data, you have to use libvirt-1.0.2 with this script:
https://bugzilla.redhat.com/attachment.cgi?id=690965&action=edit
We should probably ship this script in the examples directory of libvirt.git, and reference it more prominently from the documentation. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (4)
-
Doug Goldstein -
Eric Blake -
Peter Krempa -
Scott Sullivan