[libvirt] nodeinfo test cases for x86_64 AMD CPUs
Hi, Lately we’ve been puzzled by the nodeinfo returned for AMD 63xx based platforms so I checked out libvirt from Git and checked out the testcases in test/nodeinfodata. Currently you have 3 AMD test cases there : linux-x86_64-test3 : AMD 6172 2.1 GHz (MCM, 12 cores per package/socket, 2 numa nodes per package/socket, 1 thread/core) 4 Sockets, 8 NUMA nodes, 48 CPU cores total. linux-x86_64-test7 : AMD 6174 2.2 GHz (MCM, 12 cores per package/socket, 2 numa nodes per package/socket, 1 thread/core) 2 Sockets, 4 NUMA nodes, 24 CPU cores total. linux-x86_64-test8 : AMD 6282 SE 2.6 GHz (MCM, 8 CU(core) per package/socket, 2 numa nodes per package/socket, 2 threads/CU(core)) 4 Sockets, 8 NUMA nodes, 64 CPU cores total. However, the “expected” output from each of these are wrong I believe : % ~/libvirt/tests/nodeinfodata$ cat linux-x86_64-test{3,7,8}.expected CPUs: 48/48, MHz: 2100, Nodes: 8, Sockets: 1, Cores: 6, Threads: 1 CPUs: 24/24, MHz: 2200, Nodes: 1, Sockets: 1, Cores: 24, Threads: 1 CPUs: 64/64, MHz: 2593, Nodes: 1, Sockets: 1, Cores: 64, Threads: 1 In my opinion it should have been : CPUs: 48/48, MHz: 2100, Nodes: 8, Sockets: 4, Cores: 12, Threads: 1 CPUs: 24/24, MHz: 2200, Nodes: 4, Sockets: 2, Cores: 12, Threads: 1 CPUs: 64/64, MHz: 2593, Nodes: 8, Sockets: 4, Cores: 8, Threads: 2 Atleast that’s more in line with what tools like “lscpu” and “lstopo” would report. For example, I have a dual-socket AMD 6376 based system. lscpu reports : $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 4 Vendor ID: AuthenticAMD CPU family: 21 Model: 2 Stepping: 0 CPU MHz: 2299.820 BogoMIPS: 4600.03 Virtualization: AMD-V L1d cache: 16K L1i cache: 64K L2 cache: 2048K L3 cache: 6144K NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 NUMA node2 CPU(s): 16-23 NUMA node3 CPU(s): 24-31 virsh nodeinfo however : CPU model: x86_64 CPU(s): 32 CPU frequency: 2299 MHz CPU socket(s): 1 Core(s) per socket: 32 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 49448656 KiB Looking more closely at the code in src/nodeinfo.c:linuxNodeInfoCPUPopulate() I believe it makes some false assumptions. When iterating over the NUMA nodes, it expects that NUMA nodes can contain more than one socket, whereas on AMD systems atleast it’s the other way around (Sockets can contain more than 1 socket). Hence, the topology check at the end goes all wrong and libvirt uses just a flat topology : if ((nodeinfo->nodes * nodeinfo->sockets * nodeinfo->cores * nodeinfo->threads) != (nodeinfo->cpus + offline)) { nodeinfo->nodes = 1; nodeinfo->sockets = 1; nodeinfo->cores = nodeinfo->cpus + offline; nodeinfo->threads = 1; } If instead the “fallback” mechanism is used, which iterates over /sys/devices/system/cpu/cpu%d and finds sockets/cores/threads is used, *and* you leave nodeinfo->nodes out of the equation then it all makes sense. Iterating over entries in /sys/devices/system/node/node%d would then only be used to find nodeinfo->nodes. I have limited insight into how this would work on say an S390 system, so if my assumptions here are completely wrong please do tell :) Cheers, -- Steffen Persvold Chief Architect NumaChip, Numascale AS Tel: +47 23 16 71 88 Fax: +47 23 16 71 80 Skype: spersvold
On 22 Jan 2015, at 13:06, Steffen Persvold <sp@numascale.com> wrote:
Hi,
Lately we’ve been puzzled by the nodeinfo returned for AMD 63xx based platforms so I checked out libvirt from Git and checked out the testcases in test/nodeinfodata.
Currently you have 3 AMD test cases there :
linux-x86_64-test3 : AMD 6172 2.1 GHz (MCM, 12 cores per package/socket, 2 numa nodes per package/socket, 1 thread/core) 4 Sockets, 8 NUMA nodes, 48 CPU cores total.
linux-x86_64-test7 : AMD 6174 2.2 GHz (MCM, 12 cores per package/socket, 2 numa nodes per package/socket, 1 thread/core) 2 Sockets, 4 NUMA nodes, 24 CPU cores total.
linux-x86_64-test8 : AMD 6282 SE 2.6 GHz (MCM, 8 CU(core) per package/socket, 2 numa nodes per package/socket, 2 threads/CU(core)) 4 Sockets, 8 NUMA nodes, 64 CPU cores total.
However, the “expected” output from each of these are wrong I believe :
% ~/libvirt/tests/nodeinfodata$ cat linux-x86_64-test{3,7,8}.expected CPUs: 48/48, MHz: 2100, Nodes: 8, Sockets: 1, Cores: 6, Threads: 1 CPUs: 24/24, MHz: 2200, Nodes: 1, Sockets: 1, Cores: 24, Threads: 1 CPUs: 64/64, MHz: 2593, Nodes: 1, Sockets: 1, Cores: 64, Threads: 1
In my opinion it should have been :
CPUs: 48/48, MHz: 2100, Nodes: 8, Sockets: 4, Cores: 12, Threads: 1 CPUs: 24/24, MHz: 2200, Nodes: 4, Sockets: 2, Cores: 12, Threads: 1 CPUs: 64/64, MHz: 2593, Nodes: 8, Sockets: 4, Cores: 8, Threads: 2
Let me revise that statement a bit. Further looking at the topology in the nodeinfodata/linux-test{3,7,8} I believe the following would have been the correct interpretation of the underlying data : test3: CPUs: 48/48, MHz: 2100, Nodes: 8, Sockets: 4, Cores: 12, Threads: 1 test7: CPUs: 24/24, MHz: 2200, Nodes: 1, Sockets: 2, Cores: 12, Threads: 1 test8: CPUs: 64/64, MHz: 2593, Nodes: 8, Sockets: 4, Cores: 8, Threads: 2 The only difference is test7 which seems to be a case where the NUMA information isn’t available (but the socket numbering and cores/socket, threads/socket info is still possible to derive). In this case I believe it’s still valid to present the CPU topology as described, but with one NUMA cell only (i.e as if it was an old-fashined two-socket non-numa server). Any comments ? Cheers, -- Steffen Persvold Chief Architect NumaChip, Numascale AS Tel: +47 23 16 71 88 Fax: +47 23 16 71 80 Skype: spersvold
participants (1)
-
Steffen Persvold