[libvirt] [RFC PATCH 0/2] nodeinfo: PPC64: Fix topology and siblings info on capabilities and nodeinfo
12:32 a.m.
The nodeinfo output was fixed earlier to reflect the actual cpus available in
KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
look overcommitted when its not. The current fixes are aimed at helping the
users make better decisions on the kind of guest cpu topology that can be
supported on the given sucore_per_core setting of KVM host and also hint the
way to pin the guest vcpus efficiently.
I am planning to add some test cases once the approach is accepted.
With respect to Patch 2:
The second patch adds a new element to the cpus tag and I need your inputs on
if that is okay. Also if there is a better way. I am not sure if the existing
clients have RNG checks that might fail with the approach. Or if the checks
are not enoforced on the elements but only on the tags.
With my approach if the rng checks pass, the new element "capacity" even if
ignored by many clients would have no impact except for PPC64.
To the extent I looked at code, the siblings changes dont affect existing
libvirt functionality. Please do let me know otherwise.
---
Shivaprasad G Bhat (2):
nodeinfo: Reflect guest usable host topology on PPC64
Introduce capacity to virCapsHostNUMACellCPU to help vcpu pinning decisions
src/conf/capabilities.c | 3 +
src/conf/capabilities.h | 1
src/nodeinfo.c | 51 ++++++++++++++++++++++--
tests/vircaps2xmldata/vircaps-basic-4-4-2G.xml | 32 ++++++++-------
tests/vircaps2xmltest.c | 1
5 files changed, 67 insertions(+), 21 deletions(-)
--
Signature
12:35 a.m.
New subject: [libvirt] [RFC PATCH 1/2] nodeinfo: Reflect guest usable host topology on PPC64
Today, the topology displayed by nodeinfo/capabilities is reflecting the host
topology irrespective the subcore_per_core setting.
This is not of much use as user cannot determine what is the maximum
possible threads_per_core value he can use for a guest topology.
For subcore_per_core = 1, the guest topology can have threads_per_core=1,2,4
or 8. Same way for subcore_per_core = 2, the guest toplogy can have
threads_per_core=1,2 or 4; and for subcore_per_core = 4, guest threads_per_core
can be 1 or 2 only.
Incidentally, the topology displayed in subcore_per_core=1 is correct and is
usable today. If we can display the guest usable topology info for
subcore_per_core=2 and 4 too, then it can help making guest topology decisions.
Signed-off-by: Shivaprasad G Bhat <sbhat(a)linux.vnet.ibm.com>
---
src/nodeinfo.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/src/nodeinfo.c b/src/nodeinfo.c
index 77ea155..531e0ee 100644
--- a/src/nodeinfo.c
+++ b/src/nodeinfo.c
@@ -849,6 +849,23 @@ linuxNodeInfoCPUPopulate(const char *sysfs_prefix,
goto cleanup;
}
+ /* On PPC if host is in a valid sub core mode, the maximum supported
+ * threads_per_core for a guest is decided by the maximum
+ * threads_per_subcore.
+ * In other words on P8, if the subcores-per-core=1,
+ * the guest can have cpu topology with threads_per_core=1, 2, 4 or 8
+ * and for subcores-per-core=2, the guest can have
+ * threads_per_core=1, 2 or 4 only. Same way, for subcores-per-core=4
+ * guest can have threads_per_core=1 or 2 only.
+ * On a valid kvm host reflecddt the guest supportable topology instead
+ * of host topology.
+ */
+ if (threads_per_subcore) {
+ int subcores_per_core = nodeinfo->threads/threads_per_subcore;
+ nodeinfo->cores *= subcores_per_core;
+ nodeinfo->threads /= subcores_per_core;
+ }
+
/* Now check if the topology makes sense. There are machines that don't
* expose their real number of nodes or for example the AMD Bulldozer
* architecture that exposes their Clustered integer core modules as both
12:36 a.m.
New subject: [libvirt] [RFC PATCH 2/2] Introduce capacity to virCapsHostNUMACellCPU to help vcpu pinning decisions
The cpus tag in NUMA cell today explains core's id, socket id and the
siblings list. The sibling list is used by users to pin the vcpus of a guest
from the same guest-core to threads from the same host-core. The host
topology gives a hint of how many max siblings one might expect in the
siblings list.
For PPC64, the previous assumptions fail to make sense when the subcores are
in use. The core is split into subcores and the siblings that libvirt sees from
the sysfs are list of subcores instead of siblings. The real siblings of
subcore are in fact offline on host and brought online by the KVM scheduler
in guest context.
So, the best way to achieve efficiency in this case is to pin all the guest
threads from same guest-core to the same subcore(the primary thread which is
online). The new parameter "capacity" reflects the thread capacity of the
core/subcore and how many vcpus from the same guest-core be pinned to the
primary thread.
On PPC64, the capacity will be set to the threads_per_subcore and the
siblings list will contain "self" cpu. The capacity thus can be used to
indicate how many vcpus from guest-core can be pinned to the same subcore.
On other archs, the capacity will be set to "1" indicating to treat each core
in the sibling list to be pinned only once.
Signed-off-by: Shivaprasad G Bhat <sbhat(a)linux.vnet.ibm.com>
---
src/conf/capabilities.c | 3 +-
src/conf/capabilities.h | 1 +
src/nodeinfo.c | 34 +++++++++++++++++++++---
tests/vircaps2xmldata/vircaps-basic-4-4-2G.xml | 32 +++++++++++------------
tests/vircaps2xmltest.c | 1 +
5 files changed, 50 insertions(+), 21 deletions(-)
diff --git a/src/conf/capabilities.c b/src/conf/capabilities.c
index 86ea212..62ade2e 100644
--- a/src/conf/capabilities.c
+++ b/src/conf/capabilities.c
@@ -799,7 +799,8 @@ virCapabilitiesFormatNUMATopology(virBufferPtr buf,
virBufferAsprintf(buf, "<cpus num='%d'>\n",
cells[i]->ncpus);
virBufferAdjustIndent(buf, 2);
for (j = 0; j < cells[i]->ncpus; j++) {
- virBufferAsprintf(buf, "<cpu id='%d'",
cells[i]->cpus[j].id);
+ virBufferAsprintf(buf, "<cpu id='%d'
capacity='%d'",
+ cells[i]->cpus[j].id, cells[i]->cpus[j].capacity);
if (cells[i]->cpus[j].siblings) {
if (!(siblings = virBitmapFormat(cells[i]->cpus[j].siblings)))
diff --git a/src/conf/capabilities.h b/src/conf/capabilities.h
index 1754b13..94aa729 100644
--- a/src/conf/capabilities.h
+++ b/src/conf/capabilities.h
@@ -90,6 +90,7 @@ typedef struct _virCapsHostNUMACellCPU virCapsHostNUMACellCPU;
typedef virCapsHostNUMACellCPU *virCapsHostNUMACellCPUPtr;
struct _virCapsHostNUMACellCPU {
unsigned int id;
+ unsigned int capacity;
unsigned int socket_id;
unsigned int core_id;
virBitmapPtr siblings;
diff --git a/src/nodeinfo.c b/src/nodeinfo.c
index 531e0ee..dd2b205 100644
--- a/src/nodeinfo.c
+++ b/src/nodeinfo.c
@@ -1175,7 +1175,6 @@ linuxParseCPUmap(int max_cpuid, const char *path)
return NULL;
}
-
static virBitmapPtr
virNodeGetSiblingsList(const char *dir, int cpu_id)
{
@@ -1873,6 +1872,7 @@ nodeCapsInitNUMAFake(const char *sysfs_prefix,
if (virNodeGetCpuValue(cpupath, id, "online", 1)) {
#endif
cpus[cid].id = id;
+ cpus[cid].capacity = 1;
cpus[cid].socket_id = s;
cpus[cid].core_id = c;
if (!(cpus[cid].siblings = virBitmapNew(ncpus)))
@@ -1999,7 +1999,8 @@ nodeGetMemoryFake(unsigned long long *mem,
static int
virNodeCapsFillCPUInfo(const char *cpupath ATTRIBUTE_UNUSED,
int cpu_id ATTRIBUTE_UNUSED,
- virCapsHostNUMACellCPUPtr cpu ATTRIBUTE_UNUSED)
+ virCapsHostNUMACellCPUPtr cpu ATTRIBUTE_UNUSED,
+ int threads_per_subcore)
{
#ifdef __linux__
int tmp;
@@ -2017,9 +2018,24 @@ virNodeCapsFillCPUInfo(const char *cpupath ATTRIBUTE_UNUSED,
cpu->core_id = tmp;
- if (!(cpu->siblings = virNodeGetSiblingsList(cpupath, cpu_id)))
+ cpu->capacity = 1;
+
+ if (!threads_per_subcore &&
+ !(cpu->siblings = virNodeGetSiblingsList(cpupath, cpu_id)))
return -1;
+ /* The primary thread which is online acts in the capacity of whole core
+ * and all its thread siblings. So, if at all one wants to pin the siblings
+ * for efficiency, its better to pin the same primary thread than a
+ * primary thread from other subcore. So, show the siblings as self.
+ */
+ if (threads_per_subcore) {
+ cpu->capacity = threads_per_subcore;
+ if ((cpu->siblings = virBitmapNew(virNumaGetMaxCPUs())) == 0 ||
+ virBitmapSetBit(cpu->siblings, cpu_id) < 0)
+ return -1;
+ }
+
return 0;
#else
virReportError(VIR_ERR_NO_SUPPORT, "%s",
@@ -2123,6 +2139,7 @@ nodeCapsInitNUMA(const char *sysfs_prefix,
int cpu;
bool topology_failed = false;
int max_node;
+ int threads_per_subcore = 0;
if (virAsprintf(&cpupath, "%s/cpu", prefix) < 0)
return -1;
@@ -2132,6 +2149,14 @@ nodeCapsInitNUMA(const char *sysfs_prefix,
goto cleanup;
}
+ if ((threads_per_subcore = nodeGetThreadsPerSubcore(caps->host.arch)) < 0)
+ goto cleanup;
+
+ /* If the subcore configuration is not valid, just pretend subcores
+ * are not in use and set the thread capacity as 1*/
+ if (!nodeHasValidSubcoreConfiguration(sysfs_prefix, threads_per_subcore))
+ threads_per_subcore = 0;
+
if ((max_node = virNumaGetMaxNode()) < 0)
goto cleanup;
@@ -2151,7 +2176,8 @@ nodeCapsInitNUMA(const char *sysfs_prefix,
for (i = 0; i < virBitmapSize(cpumap); i++) {
if (virBitmapIsBitSet(cpumap, i)) {
- if (virNodeCapsFillCPUInfo(cpupath, i, cpus + cpu++) < 0) {
+ if (virNodeCapsFillCPUInfo(cpupath, i, cpus + cpu++,
+ threads_per_subcore) < 0) {
topology_failed = true;
virResetLastError();
}
diff --git a/tests/vircaps2xmldata/vircaps-basic-4-4-2G.xml
b/tests/vircaps2xmldata/vircaps-basic-4-4-2G.xml
index 8694f87..c979c4b 100644
--- a/tests/vircaps2xmldata/vircaps-basic-4-4-2G.xml
+++ b/tests/vircaps2xmldata/vircaps-basic-4-4-2G.xml
@@ -16,10 +16,10 @@
<sibling id='3' value='20'/>
</distances>
<cpus num='4'>
- <cpu id='0' socket_id='0' core_id='0'
siblings='0'/>
- <cpu id='0' socket_id='0' core_id='1'
siblings='0'/>
- <cpu id='0' socket_id='0' core_id='2'
siblings='0'/>
- <cpu id='0' socket_id='0' core_id='3'
siblings='0'/>
+ <cpu id='0' capacity='1' socket_id='0'
core_id='0' siblings='0'/>
+ <cpu id='0' capacity='1' socket_id='0'
core_id='1' siblings='0'/>
+ <cpu id='0' capacity='1' socket_id='0'
core_id='2' siblings='0'/>
+ <cpu id='0' capacity='1' socket_id='0'
core_id='3' siblings='0'/>
</cpus>
</cell>
<cell id='1'>
@@ -31,10 +31,10 @@
<sibling id='3' value='20'/>
</distances>
<cpus num='4'>
- <cpu id='1' socket_id='1' core_id='1'
siblings='1'/>
- <cpu id='1' socket_id='1' core_id='2'
siblings='1'/>
- <cpu id='1' socket_id='1' core_id='3'
siblings='1'/>
- <cpu id='1' socket_id='1' core_id='4'
siblings='1'/>
+ <cpu id='1' capacity='1' socket_id='1'
core_id='1' siblings='1'/>
+ <cpu id='1' capacity='1' socket_id='1'
core_id='2' siblings='1'/>
+ <cpu id='1' capacity='1' socket_id='1'
core_id='3' siblings='1'/>
+ <cpu id='1' capacity='1' socket_id='1'
core_id='4' siblings='1'/>
</cpus>
</cell>
<cell id='2'>
@@ -46,10 +46,10 @@
<sibling id='3' value='20'/>
</distances>
<cpus num='4'>
- <cpu id='2' socket_id='2' core_id='2'
siblings='2'/>
- <cpu id='2' socket_id='2' core_id='3'
siblings='2'/>
- <cpu id='2' socket_id='2' core_id='4'
siblings='2'/>
- <cpu id='2' socket_id='2' core_id='5'
siblings='2'/>
+ <cpu id='2' capacity='1' socket_id='2'
core_id='2' siblings='2'/>
+ <cpu id='2' capacity='1' socket_id='2'
core_id='3' siblings='2'/>
+ <cpu id='2' capacity='1' socket_id='2'
core_id='4' siblings='2'/>
+ <cpu id='2' capacity='1' socket_id='2'
core_id='5' siblings='2'/>
</cpus>
</cell>
<cell id='3'>
@@ -61,10 +61,10 @@
<sibling id='3' value='10'/>
</distances>
<cpus num='4'>
- <cpu id='3' socket_id='3' core_id='3'
siblings='3'/>
- <cpu id='3' socket_id='3' core_id='4'
siblings='3'/>
- <cpu id='3' socket_id='3' core_id='5'
siblings='3'/>
- <cpu id='3' socket_id='3' core_id='6'
siblings='3'/>
+ <cpu id='3' capacity='1' socket_id='3'
core_id='3' siblings='3'/>
+ <cpu id='3' capacity='1' socket_id='3'
core_id='4' siblings='3'/>
+ <cpu id='3' capacity='1' socket_id='3'
core_id='5' siblings='3'/>
+ <cpu id='3' capacity='1' socket_id='3'
core_id='6' siblings='3'/>
</cpus>
</cell>
</cells>
diff --git a/tests/vircaps2xmltest.c b/tests/vircaps2xmltest.c
index be2bad5..18bc490 100644
--- a/tests/vircaps2xmltest.c
+++ b/tests/vircaps2xmltest.c
@@ -51,6 +51,7 @@ buildVirCapabilities(int max_cells,
for (core_id = 0; core_id < max_cpus_in_cell; core_id++) {
cell_cpus[core_id].id = id;
+ cell_cpus[core_id].capacity = 1;
cell_cpus[core_id].socket_id = cell_id;
cell_cpus[core_id].core_id = id + core_id;
if (!(cell_cpus[core_id].siblings =
1:11 p.m.
On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
The nodeinfo output was fixed earlier to reflect the actual cpus
available in
KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
look overcommitted when its not. The current fixes are aimed at helping the
users make better decisions on the kind of guest cpu topology that can be
supported on the given sucore_per_core setting of KVM host and also hint the
way to pin the guest vcpus efficiently.
I am planning to add some test cases once the approach is accepted.
With respect to Patch 2:
The second patch adds a new element to the cpus tag and I need your inputs on
if that is okay. Also if there is a better way. I am not sure if the existing
clients have RNG checks that might fail with the approach. Or if the checks
are not enoforced on the elements but only on the tags.
With my approach if the rng checks pass, the new element "capacity" even if
ignored by many clients would have no impact except for PPC64.
To the extent I looked at code, the siblings changes dont affect existing
libvirt functionality. Please do let me know otherwise.
I've looked at your patches and I can clearly see the reasoning
behind them: basically, subcores would be presented the same way as
proper cores, so that the user doesn't have to worry about ppc64
specific details and just use the same code as x86 to optimally
arrange his guests.
As you mention, the way we report nodeinfo has been changed not to
make the host look prematurely overcommitted; however, this change
has caused a weird disconnect between nodeinfo and capabilities,
which should in theory always agree with each other.
I'm going to be way verbose in this message, stating stuff we
all know about, so that it can be used as a reference for people not
that familiar with the details and most importantly so that, if I
got anything wrong, I can be corrected :)
This is what nodeinfo currently looks like for a ppc64 host with
4 NUMA nodes, 1 socket per node, 5 cores per socket and 8 threads
per core, when subcores-per-core=1 and smt=off:
CPU(s): 160
CPU socket(s): 1
Core(s) per socket: 5
Thread(s) per core: 8
NUMA cell(s): 4
And this is part of the capabilities XML:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='5'>
<cpu id='0' socket_id='0' core_id='32'
siblings='0'/>
<cpu id='8' socket_id='0' core_id='40'
siblings='8'/>
<cpu id='16' socket_id='0' core_id='48'
siblings='16'/>
<cpu id='24' socket_id='0' core_id='96'
siblings='24'/>
<cpu id='32' socket_id='0' core_id='104'
siblings='32'/>
</cpus>
</cell>
So the information don't seem to add up: we claim that the
host has 160 online CPUs, but the NUMA topology only contains
information about 5 of them per each node, so 20 in total.
That's of course because Linux only provides us with topology
information for online CPUs, but KVM on ppc64 wants secondary
threads to be offline in the host. The secondary threads are
still used to schedule guest threads, so reporting 160 online
CPUs is correct for the purpose of planning the number of
guests based on the capacity of the host; the problem is that
the detailed NUMA topology doesn't match with that.
As a second example, here's the nodeinfo reported on the same
host when subcores-per-core=2:
CPU(s): 160
CPU socket(s): 1
Core(s) per socket: 5
Thread(s) per core: 8
NUMA cell(s): 4
and the corresponding capabilities XML:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='8' socket_id='0' core_id='40'
siblings='8,12'/>
<cpu id='12' socket_id='0' core_id='40'
siblings='8,12'/>
<cpu id='16' socket_id='0' core_id='48'
siblings='16,20'/>
<cpu id='20' socket_id='0' core_id='48'
siblings='16,20'/>
<cpu id='24' socket_id='0' core_id='96'
siblings='24,28'/>
<cpu id='28' socket_id='0' core_id='96'
siblings='24,28'/>
<cpu id='32' socket_id='0' core_id='104'
siblings='32,36'/>
<cpu id='36' socket_id='0' core_id='104'
siblings='32,36'/>
</cpus>
</cell>
Once again we only get information about the primary thread,
resulting in something that's probably very confusing unless you
know how threading works in the ppc64 world. This is, however,
exactly what the kernel exposes to userspace.
How would that change with your patches? Here's the nodeinfo:
CPU(s): 160
CPU socket(s): 1
Core(s) per socket: 10
Thread(s) per core: 4
NUMA cell(s): 4
and here's the capabilities XML:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' capacity='4' socket_id='0'
core_id='32' siblings='0'/>
<cpu id='4' capacity='4' socket_id='0'
core_id='32' siblings='4'/>
<cpu id='8' capacity='4' socket_id='0'
core_id='40' siblings='8'/>
<cpu id='12' capacity='4' socket_id='0'
core_id='40' siblings='12'/>
<cpu id='16' capacity='4' socket_id='0'
core_id='48' siblings='16'/>
<cpu id='20' capacity='4' socket_id='0'
core_id='48' siblings='20'/>
<cpu id='24' capacity='4' socket_id='0'
core_id='96' siblings='24'/>
<cpu id='28' capacity='4' socket_id='0'
core_id='96' siblings='28'/>
<cpu id='32' capacity='4' socket_id='0'
core_id='104' siblings='32'/>
<cpu id='36' capacity='4' socket_id='0'
core_id='104' siblings='36'/>
</cpus>
</cell>
So now we're basically reporting each subcore as if it were a
physical core, and once again we're only reporting information
about the primary thread of the subcore, so the output when
subcores are in use is closer to when subcores are not in use.
The additional information, in the 'capacity' attribute, can be
used by users to figure out how many vCPUs can be scheduled on
each CPU, and it can safely be ignored by existing clients; for
x86 hosts, we can just set it to 1.
The new information is more complete than it was before, and
this series certainly would help users make better guest
allocation choices. On the other hand, it doesn't really solve
the problem of nodeinfo and capabilities disagreeing with each
other, and pushes the NUMA topology reported by libvirt a
little farther from the one reported by the kernel.
It may also break some assumptions, eg. CPU 0 and 4 both have
the same value for 'core_id', so I'd expect them to be among
each other's 'siblings', but they no longer are.
I have a different proposal: since we're already altering the
NUMA topology information reported by the kernel by counting
secondary threads as online, we might as well go all the way
and rebuild the entire NUMA topology as if they were.
So the capabilities XML when subcores-per-core=1 would look
like:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='1' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='2' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='3' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='4' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='5' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='6' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='7' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='8' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='9' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='10' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='11' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='12' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='13' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='14' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='15' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='16' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='17' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='18' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='19' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='20' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='21' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='22' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='23' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='24' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='25' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='26' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='27' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='28' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='29' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='30' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='31' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='32' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='33' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='34' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='35' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='36' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='37' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='38' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='39' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
</cpus>
</cell>
and when subcores-per-core=2, it would look like:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='1' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='2' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='3' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='4' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='5' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='6' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='7' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='8' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='9' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='10' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='11' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='12' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='13' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='14' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='15' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='16' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='17' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='18' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='19' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='20' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='21' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='22' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='23' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='24' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='25' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='26' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='27' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='28' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='29' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='30' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='31' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='32' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='33' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='34' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='35' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='36' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
<cpu id='37' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
<cpu id='38' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
<cpu id='39' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
</cpus>
</cell>
which, incidentally, is pretty much the same information
the ppc64_cpu utility:
Core 0:
Subcore 0: 0* 1* 2* 3*
Subcore 1: 4* 5* 6* 7*
Core 1:
Subcore 2: 8* 9* 10* 11*
Subcore 3: 12* 13* 14* 15*
Core 2:
Subcore 4: 16* 17* 18* 19*
Subcore 5: 20* 21* 22* 23*
Core 3:
Subcore 6: 24* 25* 26* 27*
Subcore 7: 28* 29* 30* 31*
Core 4:
Subcore 8: 32* 33* 34* 35*
Subcore 9: 36* 37* 38* 39*
and libvirt itself report when subcores-per-core=2 and smt=on.
Other architectures can simply use the same value for both 'core_id'
and 'subcore_id', with no extra effort needed; the presence of a
new attribute can also be used by higher level tools to figure out
whether secondary threads are reported as online or offline.
I'm CCing both David Gibson, who might have an opinion on this due
to his work on KVM on ppc64, and Martin Polednik, because of the
same reason with s/KVM/oVirt/ :)
But of course I'd love to have as much feedback as possible :)
Cheers.
--
Andrea Bolognani
Software Engineer - Virtualization Team
8:04 p.m.
On Mon, 08 Feb 2016 20:11:41 +0100
Andrea Bolognani <abologna(a)redhat.com> wrote:
On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> The nodeinfo output was fixed earlier to reflect the actual cpus available in
> KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
> look overcommitted when its not. The current fixes are aimed at helping the
> users make better decisions on the kind of guest cpu topology that can be
> supported on the given sucore_per_core setting of KVM host and also hint the
> way to pin the guest vcpus efficiently.
>
> I am planning to add some test cases once the approach is accepted.
>
> With respect to Patch 2:
> The second patch adds a new element to the cpus tag and I need your inputs on
> if that is okay. Also if there is a better way. I am not sure if the existing
> clients have RNG checks that might fail with the approach. Or if the checks
> are not enoforced on the elements but only on the tags.
>
> With my approach if the rng checks pass, the new element "capacity" even
if
> ignored by many clients would have no impact except for PPC64.
>
> To the extent I looked at code, the siblings changes dont affect existing
> libvirt functionality. Please do let me know otherwise.
I've looked at your patches and I can clearly see the reasoning
behind them: basically, subcores would be presented the same way as
proper cores, so that the user doesn't have to worry about ppc64
specific details and just use the same code as x86 to optimally
arrange his guests.
As you mention, the way we report nodeinfo has been changed not to
make the host look prematurely overcommitted; however, this change
has caused a weird disconnect between nodeinfo and capabilities,
which should in theory always agree with each other.
I'm going to be way verbose in this message, stating stuff we
all know about, so that it can be used as a reference for people not
that familiar with the details and most importantly so that, if I
got anything wrong, I can be corrected :)
This is what nodeinfo currently looks like for a ppc64 host with
4 NUMA nodes, 1 socket per node, 5 cores per socket and 8 threads
per core, when subcores-per-core=1 and smt=off:
CPU(s): 160
CPU socket(s): 1
Core(s) per socket: 5
Thread(s) per core: 8
NUMA cell(s): 4
And this is part of the capabilities XML:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='5'>
<cpu id='0' socket_id='0' core_id='32'
siblings='0'/>
<cpu id='8' socket_id='0' core_id='40'
siblings='8'/>
<cpu id='16' socket_id='0' core_id='48'
siblings='16'/>
<cpu id='24' socket_id='0' core_id='96'
siblings='24'/>
<cpu id='32' socket_id='0' core_id='104'
siblings='32'/>
</cpus>
</cell>
So the information don't seem to add up: we claim that the
host has 160 online CPUs, but the NUMA topology only contains
information about 5 of them per each node, so 20 in total.
That's of course because Linux only provides us with topology
information for online CPUs, but KVM on ppc64 wants secondary
threads to be offline in the host. The secondary threads are
still used to schedule guest threads, so reporting 160 online
CPUs is correct for the purpose of planning the number of
guests based on the capacity of the host; the problem is that
the detailed NUMA topology doesn't match with that.
Yeah, that's rather unfortunate. We do want to list all the threads in
the capabilities, I think, since they're capable of running vcpus.
As a second example, here's the nodeinfo reported on the same
host when subcores-per-core=2:
CPU(s): 160
CPU socket(s): 1
Core(s) per socket: 5
Thread(s) per core: 8
NUMA cell(s): 4
and the corresponding capabilities XML:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='8' socket_id='0' core_id='40'
siblings='8,12'/>
<cpu id='12' socket_id='0' core_id='40'
siblings='8,12'/>
<cpu id='16' socket_id='0' core_id='48'
siblings='16,20'/>
<cpu id='20' socket_id='0' core_id='48'
siblings='16,20'/>
<cpu id='24' socket_id='0' core_id='96'
siblings='24,28'/>
<cpu id='28' socket_id='0' core_id='96'
siblings='24,28'/>
<cpu id='32' socket_id='0' core_id='104'
siblings='32,36'/>
<cpu id='36' socket_id='0' core_id='104'
siblings='32,36'/>
</cpus>
</cell>
Once again we only get information about the primary thread,
resulting in something that's probably very confusing unless you
know how threading works in the ppc64 world. This is, however,
exactly what the kernel exposes to userspace.
Ugh, yeah, that's definitely wrong. It's describing the subcores as if
they were threads.
How would that change with your patches? Here's the nodeinfo:
CPU(s): 160
CPU socket(s): 1
Core(s) per socket: 10
Thread(s) per core: 4
NUMA cell(s): 4
and here's the capabilities XML:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' capacity='4' socket_id='0'
core_id='32' siblings='0'/>
<cpu id='4' capacity='4' socket_id='0'
core_id='32' siblings='4'/>
<cpu id='8' capacity='4' socket_id='0'
core_id='40' siblings='8'/>
<cpu id='12' capacity='4' socket_id='0'
core_id='40' siblings='12'/>
<cpu id='16' capacity='4' socket_id='0'
core_id='48' siblings='16'/>
<cpu id='20' capacity='4' socket_id='0'
core_id='48' siblings='20'/>
<cpu id='24' capacity='4' socket_id='0'
core_id='96' siblings='24'/>
<cpu id='28' capacity='4' socket_id='0'
core_id='96' siblings='28'/>
<cpu id='32' capacity='4' socket_id='0'
core_id='104' siblings='32'/>
<cpu id='36' capacity='4' socket_id='0'
core_id='104' siblings='36'/>
</cpus>
</cell>
That definitely looks much better to me.
So now we're basically reporting each subcore as if it were a
physical core, and once again we're only reporting information
about the primary thread of the subcore, so the output when
subcores are in use is closer to when subcores are not in use.
The additional information, in the 'capacity' attribute, can be
used by users to figure out how many vCPUs can be scheduled on
each CPU, and it can safely be ignored by existing clients; for
x86 hosts, we can just set it to 1.
The new information is more complete than it was before, and
this series certainly would help users make better guest
allocation choices. On the other hand, it doesn't really solve
the problem of nodeinfo and capabilities disagreeing with each
other, and pushes the NUMA topology reported by libvirt a
little farther from the one reported by the kernel.
Uh.. I don't really see how nodeinfo and capabilities disagree here.
Now how the topology differs from the kernel.
It may also break some assumptions, eg. CPU 0 and 4 both have
the same value for 'core_id', so I'd expect them to be among
each other's 'siblings', but they no longer are.
Ah, yes, that's wrong. With this setup the core_id should be set to
the id of the subcore's first thread, rather than the physical core's
first thread.
I have a different proposal: since we're already altering the
NUMA topology information reported by the kernel by counting
secondary threads as online, we might as well go all the way
and rebuild the entire NUMA topology as if they were.
So the capabilities XML when subcores-per-core=1 would look
like:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
I don't think adding a subcore_id is a good idea. Because it's only
ever likely to mean something on ppc64, tools are qlikely to just
ignore it and use the core_id. Most of the time that will be wrong:
behaviorally subcores act like cores in almost all regards.
That could be worked around by instead having core_id give the subcore
address and adding a new "supercore_id" or "core_group_id" or
something.
But frankly, I don't think there's actually much point exposing the
physical topology in addition to the logical (subcore) topology. Yes,
subcores will have different performance characteristics to real cores
which will be relevant in some situations. But if you're manually
setting the host's subcore mode then you're already into the realm of
manually tweaking parameters based on knowledge of your system and
workload. Basically I don't see anything upper layers would do with
the subcore vs. core information that isn't likely to be overriden by
manual tweaks in any case where you're setting the subcore mode at all.
Bear in mind that now that we have dynamic split core merged, I'm not
sure there are *any* realistic use cases for using manual subcore
splitting.
<cpu id='1' socket_id='0'
core_id='32' subcore_id='0' siblings='0-7'/>
<cpu id='2' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='3' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='4' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='5' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='6' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='7' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='8' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='9' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='10' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='11' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='12' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='13' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='14' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='15' socket_id='0' core_id='40'
subcore_id='1' siblings='8-15'/>
<cpu id='16' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='17' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='18' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='19' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='20' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='21' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='22' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='23' socket_id='0' core_id='48'
subcore_id='2' siblings='16-23'/>
<cpu id='24' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='25' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='26' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='27' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='28' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='29' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='30' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='31' socket_id='0' core_id='96'
subcore_id='3' siblings='24-31'/>
<cpu id='32' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='33' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='34' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='35' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='36' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='37' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='38' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
<cpu id='39' socket_id='0' core_id='104'
subcore_id='4' siblings='32-39'/>
</cpus>
</cell>
and when subcores-per-core=2, it would look like:
<cells num='4'>
<cell id='0'>
<memory unit='KiB'>67108864</memory>
<pages unit='KiB' size='64'>1048576</pages>
<pages unit='KiB' size='16384'>0</pages>
<distances>
<sibling id='0' value='10'/>
<sibling id='1' value='20'/>
<sibling id='16' value='40'/>
<sibling id='17' value='40'/>
</distances>
<cpus num='10'>
<cpu id='0' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='1' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='2' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
<cpu id='3' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
In this case, I think siblings should be 0-3. Again subcores act like
cores for most purposes.
<cpu id='4' socket_id='0'
core_id='32' subcore_id='1' siblings='0-7'/>
<cpu id='5' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='6' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='7' socket_id='0' core_id='32'
subcore_id='1' siblings='0-7'/>
<cpu id='8' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='9' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='10' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='11' socket_id='0' core_id='40'
subcore_id='2' siblings='8-15'/>
<cpu id='12' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='13' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='14' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='15' socket_id='0' core_id='40'
subcore_id='3' siblings='8-15'/>
<cpu id='16' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='17' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='18' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='19' socket_id='0' core_id='48'
subcore_id='4' siblings='16-23'/>
<cpu id='20' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='21' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='22' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='23' socket_id='0' core_id='48'
subcore_id='5' siblings='16-23'/>
<cpu id='24' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='25' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='26' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='27' socket_id='0' core_id='96'
subcore_id='6' siblings='24-31'/>
<cpu id='28' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='29' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='30' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='31' socket_id='0' core_id='96'
subcore_id='7' siblings='24-31'/>
<cpu id='32' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='33' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='34' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='35' socket_id='0' core_id='104'
subcore_id='8' siblings='32-39'/>
<cpu id='36' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
<cpu id='37' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
<cpu id='38' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
<cpu id='39' socket_id='0' core_id='104'
subcore_id='9' siblings='32-39'/>
</cpus>
</cell>
which, incidentally, is pretty much the same information
the ppc64_cpu utility:
Core 0:
Subcore 0: 0* 1* 2* 3*
Subcore 1: 4* 5* 6* 7*
Core 1:
Subcore 2: 8* 9* 10* 11*
Subcore 3: 12* 13* 14* 15*
Core 2:
Subcore 4: 16* 17* 18* 19*
Subcore 5: 20* 21* 22* 23*
Core 3:
Subcore 6: 24* 25* 26* 27*
Subcore 7: 28* 29* 30* 31*
Core 4:
Subcore 8: 32* 33* 34* 35*
Subcore 9: 36* 37* 38* 39*
and libvirt itself report when subcores-per-core=2 and smt=on.
Other architectures can simply use the same value for both 'core_id'
and 'subcore_id', with no extra effort needed; the presence of a
new attribute can also be used by higher level tools to figure out
whether secondary threads are reported as online or offline.
I'm CCing both David Gibson, who might have an opinion on this due
to his work on KVM on ppc64, and Martin Polednik, because of the
same reason with s/KVM/oVirt/ :)
But of course I'd love to have as much feedback as possible :)
So, as noted, I actually prefer Shivaprasad's original approach of
treating subcores as cores. The implementation of that does need some
adjustment as noted above, basically treating subcores as cores even
more universally than the current patches do.
Basically I see manually setting the subcores-per-core as telling the
system you want it treated as having more cores. So libvirt shouldn't
decide it knows better and report physical cores instead.
Tangentially related is the question of how to deal with threads which
are offline in the host but can be used for vcpus, which appears with
or without subcores.
I have no very strong opinion between the options of (1) adding <cpu>
entries for the offline threads (whose siblings should be set based on
the subcore) or (2) adding a capacity tag to the online threads
indicating that you can put more vcpus on them that the host online
thread count indicates.
(1) is more likely to do the right thing with existing tools, but (2)
is more accurately expressive.
--
David Gibson <dgibson(a)redhat.com>
Senior Software Engineer, Virtualization, Red Hat
10:40 a.m.
On Tue, 2016-02-16 at 13:04 +1100, David Gibson wrote:
> So the information don't seem to add up: we claim that the
> host has 160 online CPUs, but the NUMA topology only contains
> information about 5 of them per each node, so 20 in total.
>
> That's of course because Linux only provides us with topology
> information for online CPUs, but KVM on ppc64 wants secondary
> threads to be offline in the host. The secondary threads are
> still used to schedule guest threads, so reporting 160 online
> CPUs is correct for the purpose of planning the number of
> guests based on the capacity of the host; the problem is that
> the detailed NUMA topology doesn't match with that.
Yeah, that's rather unfortunate. We do want to list all the threads in
the capabilities, I think, since they're capable of running vcpus.
There's a problem with that, though, that I didn't think about
when writing the original message. See below.
> The new information is more complete than it was before, and
> this series certainly would help users make better guest
> allocation choices. On the other hand, it doesn't really solve
> the problem of nodeinfo and capabilities disagreeing with each
> other, and pushes the NUMA topology reported by libvirt a
> little farther from the one reported by the kernel.
Uh.. I don't really see how nodeinfo and capabilities disagree here.
Because nodeinfo tell us that the host has 160 online CPUs, while
capabilities tell us that only 40 CPUs are actually online.
Just to clarify, since the naming is not very explicit: nodeinfo
reports *online* CPUs only, and each <cpu> element in capabilities
is supposed to represent an *online* CPU. Offline CPUs are not
listed or counted anywhere.
Now how the topology differs from the kernel.
Because the kernel reports physical information, so online CPUs
that are in the same physical core (as evidenced by the core_id
value) are counted as siblings regardless of the subcore
configuration.
> It may also break some assumptions, eg. CPU 0 and 4 both have
> the same value for 'core_id', so I'd expect them to be among
> each other's 'siblings', but they no longer are.
Ah, yes, that's wrong. With this setup the core_id should be set to
the id of the subcore's first thread, rather than the physical core's
first thread.
This would be a pretty wild departure from what the kernel is
exposing to userspace.
> I have a different proposal: since we're already altering
the
> NUMA topology information reported by the kernel by counting
> secondary threads as online, we might as well go all the way
> and rebuild the entire NUMA topology as if they were.
>
> So the capabilities XML when subcores-per-core=1 would look
> like:
>
> <cells num='4'>
> <cell id='0'>
> <memory unit='KiB'>67108864</memory>
> <pages unit='KiB' size='64'>1048576</pages>
> <pages unit='KiB' size='16384'>0</pages>
> <distances>
> <sibling id='0' value='10'/>
> <sibling id='1' value='20'/>
> <sibling id='16' value='40'/>
> <sibling id='17' value='40'/>
> </distances>
> <cpus num='10'>
> <cpu id='0' socket_id='0' core_id='32'
subcore_id='0' siblings='0-7'/>
I don't think adding a subcore_id is a good idea. Because it's only
ever likely to mean something on ppc64, tools are qlikely to just
ignore it and use the core_id.
Fair enough, and something we definitely need to take into account.
Most of the time that will be wrong:
behaviorally subcores act like cores in almost all regards.
That could be worked around by instead having core_id give the subcore
address and adding a new "supercore_id" or "core_group_id" or
something.
Well, those would be just as likely to be ignored by tools as
subcore_id and capacity, wouldn't they? :)
But frankly, I don't think there's actually much point
exposing the
physical topology in addition to the logical (subcore) topology. Yes,
subcores will have different performance characteristics to real cores
which will be relevant in some situations. But if you're manually
setting the host's subcore mode then you're already into the realm of
manually tweaking parameters based on knowledge of your system and
workload. Basically I don't see anything upper layers would do with
the subcore vs. core information that isn't likely to be overriden by
manual tweaks in any case where you're setting the subcore mode at all.
Bear in mind that now that we have dynamic split core merged, I'm not
sure there are *any* realistic use cases for using manual subcore
splitting.
As far as I know, and of course correct me if I'm wrong, dynamic split
cores are a way to avoid heavy performance drops when the host is
overcommitted, but it has some overhead of its own because of the need
to split the core when entering KVM and restoring the previous
configuration when exiting it.
Splitting cores manually, on the other hand, is less flexible but
gives the admin complete control over resource allocations and has
no overhead, which makes it critical for fine performance tuning.
So unless my understanding is off, I believe we need to expose the
topology in a way that enables both use cases.
So, as noted, I actually prefer Shivaprasad's original approach
of
treating subcores as cores. The implementation of that does need some
adjustment as noted above, basically treating subcores as cores even
more universally than the current patches do.
Basically I see manually setting the subcores-per-core as telling the
system you want it treated as having more cores. So libvirt shouldn't
decide it knows better and report physical cores instead.
It's not really a case of libvirt thinking it knows better, it's more
a case of the kernel always reporting information about the physical
topology and x86 not having AFAIK any configuration where physical
and logical topology disagree, whereas that's the case on ppc64 as
soon as you split a core.
Tangentially related is the question of how to deal with threads
which
are offline in the host but can be used for vcpus, which appears with
or without subcores.
This is exactly the issue I was thinking about at the beginning of
the message: if I see a <cpu> element that means the CPU is online,
and I expect to be able to pin a vCPU to it, but that would no
longer be the case if we started adding <cpu> elements for CPUs
that are offline in the host - any attempt to pin vCPUs to such
CPUs would fail.
I have no very strong opinion between the options of (1) adding
<cpu>
entries for the offline threads (whose siblings should be set based on
the subcore) or (2) adding a capacity tag to the online threads
indicating that you can put more vcpus on them that the host online
thread count indicates.
(1) is more likely to do the right thing with existing tools, but (2)
is more accurately expressive.
I kinda think it would be the other way around, as I expect existing
tools to make some of the assumptions described above.
So after discussing for a while with Shivaprasad on IRC, reading
your reply and thinking about it some more, I'm starting to think
that we should report the same information as the kernel whenever
possible, and enhance it with hints such as capacity for tools
and admins, but carefully avoid being too clever for our own good.
We can paper over most of the differences between x86 and ppc64,
but not all of them, I believe. Documentation is key here, and
there's definitely room for improvement both when it comes to
reasonable assumptions and ppc64 specific quirks - this is
something I'm planning to address.
So my proposal is to drop patch 1/2 entirely and rework patch 2/2
to just add the capacity attribute (maybe with a more descriptive
name?), without touching the list of siblings or any other
information retrieved from the kernel.
Cheers.
--
Andrea Bolognani
Software Engineer - Virtualization Team
1:48 p.m.
On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
The nodeinfo output was fixed earlier to reflect the actual cpus
available in
KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
look overcommitted when its not. The current fixes are aimed at helping the
users make better decisions on the kind of guest cpu topology that can be
supported on the given sucore_per_core setting of KVM host and also hint the
way to pin the guest vcpus efficiently.
I am planning to add some test cases once the approach is accepted.
With respect to Patch 2:
The second patch adds a new element to the cpus tag and I need your inputs on
if that is okay. Also if there is a better way. I am not sure if the existing
clients have RNG checks that might fail with the approach. Or if the checks
are not enoforced on the elements but only on the tags.
With my approach if the rng checks pass, the new element "capacity" even if
ignored by many clients would have no impact except for PPC64.
To the extent I looked at code, the siblings changes dont affect existing
libvirt functionality. Please do let me know otherwise.
So, I've been going through this old thread trying to figure out
a way to improve the status quo. I'd like to collect as much
feedback as possible, especially from people who have worked in
this area of libvirt before or have written tools based on it.
As hinted above, this series is really trying to address two
different issue, and I think it's helpful to reason about them
separately.
** Guest threads limit **
My dual-core laptop will happily run a guest configured with
<cpu>
<topology sockets='1' cores='1' threads='128'/>
</cpu>
but POWER guests are limited to 8/subcores_per_core threads.
We need to report this information to the user somehow, and
I can't see an existing place where it would fit nicely. We
definitely don't want to overload the meaning of an existing
element/attribute with this. It should also only appear in
the (dom)capabilities XML of ppc64 hosts.
I don't think this is too problematic or controversial, we
just need to pick a nice place to display this information.
** Efficient guest topology **
To achieve optimal performance, you want to match guest
threads with host threads.
On x86, you can choose suitable host threads by looking at
the capabilities XML: the presence of elements like
<cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
<cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
means you should configure your guest to use
<vcpu placement='static' cpuset='2-3'>2</vcpu>
<cpu>
<topology sockets='1' cores='1' threads='2'/>
</cpu>
Notice how siblings can be found either looking at the
attribute with the same name, or by matching them using the
value of the core_id attribute. Also notice how you are
supposed to pin as many vCPUs as the number of elements in
the cpuset - one guest thread per host thread.
On POWER, this gets much trickier: only the *primary* thread
of each (sub)core appears to be online in the host, but all
threads can actually have a vCPU running on them. So
<cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
which is what you'd get with subcores_per_core=2, is very
confusing.
The optimal guest topology in this case would be
<vcpu placement='static' cpuset='4'>4</vcpu>
<cpu>
<topology sockets='1' cores='1' threads='4'/>
</cpu>
but neither approaches mentioned above work to figure out the
correct value for the cpuset attribute.
In this case, a possible solution would be to alter the values
of the core_id and siblings attribute such that both would be
the same as the id attribute, which would naturally make both
approaches described above work.
Additionaly, a new attribute would be introduced to serve as
a multiplier for the "one guest thread per host thread" rule
mentioned earlier: the resulting XML would look like
<cpu id='0' socket_id='0' core_id='0' siblings='0'
capacity='4'/>
<cpu id='4' socket_id='0' core_id='4' siblings='4'
capacity='4'/>
which contains all the information needed to build the right
guest topology. The capacity attribute would have value 1 on
all architectures except for ppc64.
We could arguably use the capacity attribute to cover the
use case described in the first part as well, by declaring that
any value other than 1 means there's a limit to the number of
threads a guest core can have. I think doing so has the
potential to produce much grief in the future, so I'd rather
keep them separate - even if it means inventing a new element.
It's been also proposed to add a physical_core_id attribute,
which would contain the real core id and allow tools to figure
out which subcores belong to the same core - it would be the
same as core_id for all other architectures and for ppc64
when subcores_per_core=1. It's not clear whether having this
attribute would be useful or just confusing.
This is all I have for now. Please let me know what you think
about it.
--
Andrea Bolognani
Software Engineer - Virtualization Team
4:59 p.m.
On 05/05/2016 02:48 PM, Andrea Bolognani wrote:
On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> The nodeinfo output was fixed earlier to reflect the actual cpus available in
> KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
> look overcommitted when its not. The current fixes are aimed at helping the
> users make better decisions on the kind of guest cpu topology that can be
> supported on the given sucore_per_core setting of KVM host and also hint the
> way to pin the guest vcpus efficiently.
>
> I am planning to add some test cases once the approach is accepted.
>
> With respect to Patch 2:
> The second patch adds a new element to the cpus tag and I need your inputs on
> if that is okay. Also if there is a better way. I am not sure if the existing
> clients have RNG checks that might fail with the approach. Or if the checks
> are not enoforced on the elements but only on the tags.
>
> With my approach if the rng checks pass, the new element "capacity" even
if
> ignored by many clients would have no impact except for PPC64.
>
> To the extent I looked at code, the siblings changes dont affect existing
> libvirt functionality. Please do let me know otherwise.
So, I've been going through this old thread trying to figure out
a way to improve the status quo. I'd like to collect as much
feedback as possible, especially from people who have worked in
this area of libvirt before or have written tools based on it.
As hinted above, this series is really trying to address two
different issue, and I think it's helpful to reason about them
separately.
** Guest threads limit **
My dual-core laptop will happily run a guest configured with
<cpu>
<topology sockets='1' cores='1' threads='128'/>
</cpu>
but POWER guests are limited to 8/subcores_per_core threads.
How is it limited? Does something explicitly fail (libvirt, qemu, guest OS)?
Or are the threads just not usable in the VM
Is it specific to PPC64 KVM, or PPC64 emulated as well?
We need to report this information to the user somehow, and
I can't see an existing place where it would fit nicely. We
definitely don't want to overload the meaning of an existing
element/attribute with this. It should also only appear in
the (dom)capabilities XML of ppc64 hosts.
I don't think this is too problematic or controversial, we
just need to pick a nice place to display this information.
** Efficient guest topology **
To achieve optimal performance, you want to match guest
threads with host threads.
On x86, you can choose suitable host threads by looking at
the capabilities XML: the presence of elements like
<cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
<cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
means you should configure your guest to use
<vcpu placement='static' cpuset='2-3'>2</vcpu>
<cpu>
<topology sockets='1' cores='1' threads='2'/>
</cpu>
Notice how siblings can be found either looking at the
attribute with the same name, or by matching them using the
value of the core_id attribute. Also notice how you are
supposed to pin as many vCPUs as the number of elements in
the cpuset - one guest thread per host thread.
Ahh, I see that threads are implicitly reported by the fact that socket_id and
core_id are identical across the different cpu ids... that took me a couple
minutes :)
On POWER, this gets much trickier: only the *primary* thread
of each (sub)core appears to be online in the host, but all
threads can actually have a vCPU running on them. So
<cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
which is what you'd get with subcores_per_core=2, is very
confusing.
Okay, this bit took me _more_ than a couple minutes. Is this saying topology of
socket #0
core #32
subcore #1
cpu id='0' thread #1
cpu id='1' thread #2 (offline)
cpu id='2' thread #3 (offline)
cpu id='3' thread #4 (offline)
subcore #2
cpu id='4' thread #1
cpu id='5' thread #2 (offline)
cpu id='6' thread #3 (offline)
cpu id='7' thread #4 (offline)
...
what would the hypothetical physical_core_id value look like in that example?
The optimal guest topology in this case would be
<vcpu placement='static' cpuset='4'>4</vcpu>
<cpu>
<topology sockets='1' cores='1' threads='4'/>
</cpu>
So when we pin to logical CPU #4, ppc KVM is smart enough to see that it's a
subcore thread, will then make use of the offline threads in the same subcore?
Or does libvirt do anything fancy to facilitate this case?
but neither approaches mentioned above work to figure out the
correct value for the cpuset attribute.
In this case, a possible solution would be to alter the values
of the core_id and siblings attribute such that both would be
the same as the id attribute, which would naturally make both
approaches described above work.
Additionaly, a new attribute would be introduced to serve as
a multiplier for the "one guest thread per host thread" rule
mentioned earlier: the resulting XML would look like
<cpu id='0' socket_id='0' core_id='0' siblings='0'
capacity='4'/>
<cpu id='4' socket_id='0' core_id='4' siblings='4'
capacity='4'/>
which contains all the information needed to build the right
guest topology. The capacity attribute would have value 1 on
all architectures except for ppc64.
capacity is pretty generic sounding... not sure if that's good or not in this
case. maybe thread_capacity?
We could arguably use the capacity attribute to cover the
use case described in the first part as well, by declaring that
any value other than 1 means there's a limit to the number of
threads a guest core can have. I think doing so has the
potential to produce much grief in the future, so I'd rather
keep them separate - even if it means inventing a new element.
It's been also proposed to add a physical_core_id attribute,
which would contain the real core id and allow tools to figure
out which subcores belong to the same core - it would be the
same as core_id for all other architectures and for ppc64
when subcores_per_core=1. It's not clear whether having this
attribute would be useful or just confusing.
IMO it seems like something worth adding since it is a pertinent piece of the
topology, even if there isn't a clear programmatic use for it yet.
This is all I have for now. Please let me know what you think
about it.
FWIW virt-manager basically doesn't consume the host topology XML, so there's
no concern there.
A quick grep seems to indicate that both nova (openstack) and vdsm
(ovirt/rhev) _do_ consume this XML for their numa magic (git grep sibling),
but I can't speak to the details of how it's consumed.
- Cole
4:49 a.m.
On Tue, 2016-05-10 at 17:59 -0400, Cole Robinson wrote:
On 05/05/2016 02:48 PM, Andrea Bolognani wrote:
> On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
>
> ** Guest threads limit **
>
> My dual-core laptop will happily run a guest configured with
>
> <cpu>
> <topology sockets='1' cores='1' threads='128'/>
> </cpu>
>
> but POWER guests are limited to 8/subcores_per_core threads.
How is it limited? Does something explicitly fail (libvirt, qemu, guest OS)?
Or are the threads just not usable in the VM
Is it specific to PPC64 KVM, or PPC64 emulated as well?
QEMU fails with errors like
qemu-kvm: Cannot support more than 8 threads on PPC with KVM
qemu-kvm: Cannot support more than 1 threads on PPC with TCG
depending on the guest type.
> We need to report this information to the user somehow, and
> I can't see an existing place where it would fit nicely. We
> definitely don't want to overload the meaning of an existing
> element/attribute with this. It should also only appear in
> the (dom)capabilities XML of ppc64 hosts.
>
> I don't think this is too problematic or controversial, we
> just need to pick a nice place to display this information.
Adding to the above: we already have
<vcpu max='...'/>
in the domcapabilities XML, and there was some recent
discussion about improving the information reported there.
Possibly a good match?
> ** Efficient guest topology **
>
> To achieve optimal performance, you want to match guest
> threads with host threads.
>
> On x86, you can choose suitable host threads by looking at
> the capabilities XML: the presence of elements like
>
> <cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
> <cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
>
> means you should configure your guest to use
>
> <vcpu placement='static' cpuset='2-3'>2</vcpu>
> <cpu>
> <topology sockets='1' cores='1' threads='2'/>
> </cpu>
>
> Notice how siblings can be found either looking at the
> attribute with the same name, or by matching them using the
> value of the core_id attribute. Also notice how you are
> supposed to pin as many vCPUs as the number of elements in
> the cpuset - one guest thread per host thread.
Ahh, I see that threads are implicitly reported by the fact that socket_id and
core_id are identical across the different cpu ids... that took me a couple
minutes :)
Yup :)
thread_siblings_list, the sysfs topology file we read to fill
in the 'siblings' attribute, actually contains the internal
information the kernel has gathered by matching socket_id (aka
physical_package_id in sysfs) and core_id[1].
> On POWER, this gets much trickier: only the *primary* thread
> of each (sub)core appears to be online in the host, but all
> threads can actually have a vCPU running on them. So
>
> <cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
> <cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
>
> which is what you'd get with subcores_per_core=2, is very
> confusing.
Okay, this bit took me _more_ than a couple minutes. Is this saying topology of
socket #0
core #32
subcore #1
cpu id='0' thread #1
cpu id='1' thread #2 (offline)
cpu id='2' thread #3 (offline)
cpu id='3' thread #4 (offline)
subcore #2
cpu id='4' thread #1
cpu id='5' thread #2 (offline)
cpu id='6' thread #3 (offline)
cpu id='7' thread #4 (offline)
...
what would the hypothetical physical_core_id value look like in that example?
physical_core_id would be 32 for all of the above - it would
just be the very value of core_id the kernel reads from the
hardware and reports through sysfs.
The tricky bit is that, when subcores are in use, core_id and
physical_core_id would not match. They will always match on
architectures that lack the concept of subcores, though.
> The optimal guest topology in this case would be
>
> <vcpu placement='static' cpuset='4'>4</vcpu>
> <cpu>
> <topology sockets='1' cores='1' threads='4'/>
> </cpu>
So when we pin to logical CPU #4, ppc KVM is smart enough to see that it's a
subcore thread, will then make use of the offline threads in the same subcore?
Or does libvirt do anything fancy to facilitate this case?
My understanding is that libvirt shouldn't have to do anything
to pass the hint to kvm, but David will have the authoritative
answer here.
> but neither approaches mentioned above work to figure out the
> correct value for the cpuset attribute.
>
> In this case, a possible solution would be to alter the values
> of the core_id and siblings attribute such that both would be
> the same as the id attribute, which would naturally make both
> approaches described above work.
>
> Additionaly, a new attribute would be introduced to serve as
> a multiplier for the "one guest thread per host thread" rule
> mentioned earlier: the resulting XML would look like
>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0' capacity='4'/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4' capacity='4'/>
>
> which contains all the information needed to build the right
> guest topology. The capacity attribute would have value 1 on
> all architectures except for ppc64.
capacity is pretty generic sounding... not sure if that's good or not in this
case. maybe thread_capacity?
Yeah, I'm not in love with the name either, but I've been unable
to come up with a better one myself. thread_capacity might be a
tiny bit better, but in the end I think there's little chance
we'll be able to find a good, short name for "you can pin this
number of guest threads to this host thread" - let's pick
something not horrible and document the heck out of it.
> We could arguably use the capacity attribute to cover the
> use case described in the first part as well, by declaring that
> any value other than 1 means there's a limit to the number of
> threads a guest core can have. I think doing so has the
> potential to produce much grief in the future, so I'd rather
> keep them separate - even if it means inventing a new element.
>
> It's been also proposed to add a physical_core_id attribute,
> which would contain the real core id and allow tools to figure
> out which subcores belong to the same core - it would be the
> same as core_id for all other architectures and for ppc64
> when subcores_per_core=1. It's not clear whether having this
> attribute would be useful or just confusing.
IMO it seems like something worth adding since it is a pertinent piece of the
topology, even if there isn't a clear programmatic use for it yet.
It is a piece of information that we would not be reporting,
that much is clear. However, as mentioned above, I'm afraid it
might make things more confusing, especially for architectures
that do not have subcores - basically all of them.
So maybe we should only add this information once its usefulness
has been proven.
> This is all I have for now. Please let me know what you think
> about it.
FWIW virt-manager basically doesn't consume the host topology XML, so there's
no concern there.
That's good to know :)
A quick grep seems to indicate that both nova (openstack) and vdsm
(ovirt/rhev) _do_ consume this XML for their numa magic (git grep sibling),
but I can't speak to the details of how it's consumed.
We won't know whether the proposal is actually sensible until
David weighs in, but I'm adding Martin back in the loop so
we can maybe give us the oVirt angle in the meantime.
Thanks for sharing your thoughts!
[1] https://www.kernel.org/doc/Documentation/cputopology.txt
--
Andrea Bolognani
Software Engineer - Virtualization Team
1:08 a.m.
On Tue, 17 May 2016 11:49:22 +0200
Andrea Bolognani <abologna(a)redhat.com> wrote:
On Tue, 2016-05-10 at 17:59 -0400, Cole Robinson wrote:
> On 05/05/2016 02:48 PM, Andrea Bolognani wrote:
> > On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> >
> > ** Guest threads limit **
> >
> > My dual-core laptop will happily run a guest configured with
> >
> > <cpu>
> > <topology sockets='1' cores='1'
threads='128'/>
> > </cpu>
> >
> > but POWER guests are limited to 8/subcores_per_core threads.
>
> How is it limited? Does something explicitly fail (libvirt, qemu, guest OS)?
> Or are the threads just not usable in the VM
>
> Is it specific to PPC64 KVM, or PPC64 emulated as well?
QEMU fails with errors like
qemu-kvm: Cannot support more than 8 threads on PPC with KVM
qemu-kvm: Cannot support more than 1 threads on PPC with TCG
depending on the guest type.
Note that in a sense the two errors come about for different reasons.
On Power, to a much greater degree than x86, threads on the same core
have observably different behaviour from threads on different cores.
Because of that, there's no reasonable way for KVM to present more
guest threads-per-core than there are host threads-per-core.
The limit of 1 thread on TCG is simply because no-one's ever bothered
to implement SMT emulation in qemu.
> > We need to report this information to the user somehow,
and
> > I can't see an existing place where it would fit nicely. We
> > definitely don't want to overload the meaning of an existing
> > element/attribute with this. It should also only appear in
> > the (dom)capabilities XML of ppc64 hosts.
> >
> > I don't think this is too problematic or controversial, we
> > just need to pick a nice place to display this information.
Adding to the above: we already have
<vcpu max='...'/>
in the domcapabilities XML, and there was some recent
discussion about improving the information reported there.
Possibly a good match?
> > ** Efficient guest topology **
> >
> > To achieve optimal performance, you want to match guest
> > threads with host threads.
> >
> > On x86, you can choose suitable host threads by looking at
> > the capabilities XML: the presence of elements like
> >
> > <cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
> > <cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
> >
> > means you should configure your guest to use
> >
> > <vcpu placement='static' cpuset='2-3'>2</vcpu>
> > <cpu>
> > <topology sockets='1' cores='1'
threads='2'/>
> > </cpu>
> >
> > Notice how siblings can be found either looking at the
> > attribute with the same name, or by matching them using the
> > value of the core_id attribute. Also notice how you are
> > supposed to pin as many vCPUs as the number of elements in
> > the cpuset - one guest thread per host thread.
>
> Ahh, I see that threads are implicitly reported by the fact that socket_id and
> core_id are identical across the different cpu ids... that took me a couple
> minutes :)
Yup :)
thread_siblings_list, the sysfs topology file we read to fill
in the 'siblings' attribute, actually contains the internal
information the kernel has gathered by matching socket_id (aka
physical_package_id in sysfs) and core_id[1].
> > On POWER, this gets much trickier: only the *primary* thread
> > of each (sub)core appears to be online in the host, but all
> > threads can actually have a vCPU running on them. So
> >
> > <cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
> > <cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
> >
> > which is what you'd get with subcores_per_core=2, is very
> > confusing.
>
> Okay, this bit took me _more_ than a couple minutes. Is this saying topology of
>
> socket #0
> core #32
> subcore #1
> cpu id='0' thread #1
> cpu id='1' thread #2 (offline)
> cpu id='2' thread #3 (offline)
> cpu id='3' thread #4 (offline)
> subcore #2
> cpu id='4' thread #1
> cpu id='5' thread #2 (offline)
> cpu id='6' thread #3 (offline)
> cpu id='7' thread #4 (offline)
> ...
>
> what would the hypothetical physical_core_id value look like in that example?
physical_core_id would be 32 for all of the above - it would
just be the very value of core_id the kernel reads from the
hardware and reports through sysfs.
The tricky bit is that, when subcores are in use, core_id and
physical_core_id would not match. They will always match on
architectures that lack the concept of subcores, though.
Yeah, I'm still not terribly convinced that we should even be
presenting physical core info instead of *just* logical core info. If
you care that much about physical core topology, you probably
shouldn't be running your system in subcore mode.
> > The optimal guest topology in this case would be
> >
> > <vcpu placement='static' cpuset='4'>4</vcpu>
> > <cpu>
> > <topology sockets='1' cores='1'
threads='4'/>
> > </cpu>
>
> So when we pin to logical CPU #4, ppc KVM is smart enough to see that it's a
> subcore thread, will then make use of the offline threads in the same subcore?
> Or does libvirt do anything fancy to facilitate this case?
My understanding is that libvirt shouldn't have to do anything
to pass the hint to kvm, but David will have the authoritative
answer here.
Um.. I'm not totally certain. It will be one of two things:
a) you just bind the guest thread to the representative host thread
b) you bind the guest thread to a cpumask with all of the host
threads on the relevant (sub)core - including the offline host
threads
I'll try to figure out which one it is.
> > but neither approaches mentioned above work to figure out
the
> > correct value for the cpuset attribute.
> >
> > In this case, a possible solution would be to alter the values
> > of the core_id and siblings attribute such that both would be
> > the same as the id attribute, which would naturally make both
> > approaches described above work.
> >
> > Additionaly, a new attribute would be introduced to serve as
> > a multiplier for the "one guest thread per host thread" rule
> > mentioned earlier: the resulting XML would look like
> >
> > <cpu id='0' socket_id='0' core_id='0'
siblings='0' capacity='4'/>
> > <cpu id='4' socket_id='0' core_id='4'
siblings='4' capacity='4'/>
> >
> > which contains all the information needed to build the right
> > guest topology. The capacity attribute would have value 1 on
> > all architectures except for ppc64.
>
> capacity is pretty generic sounding... not sure if that's good or not in this
> case. maybe thread_capacity?
Yeah, I'm not in love with the name either, but I've been unable
to come up with a better one myself. thread_capacity might be a
tiny bit better, but in the end I think there's little chance
we'll be able to find a good, short name for "you can pin this
number of guest threads to this host thread" - let's pick
something not horrible and document the heck out of it.
> > We could arguably use the capacity attribute to cover the
> > use case described in the first part as well, by declaring that
> > any value other than 1 means there's a limit to the number of
> > threads a guest core can have. I think doing so has the
> > potential to produce much grief in the future, so I'd rather
> > keep them separate - even if it means inventing a new element.
> >
> > It's been also proposed to add a physical_core_id attribute,
> > which would contain the real core id and allow tools to figure
> > out which subcores belong to the same core - it would be the
> > same as core_id for all other architectures and for ppc64
> > when subcores_per_core=1. It's not clear whether having this
> > attribute would be useful or just confusing.
>
> IMO it seems like something worth adding since it is a pertinent piece of the
> topology, even if there isn't a clear programmatic use for it yet.
It is a piece of information that we would not be reporting,
that much is clear. However, as mentioned above, I'm afraid it
might make things more confusing, especially for architectures
that do not have subcores - basically all of them.
So maybe we should only add this information once its usefulness
has been proven.
> > This is all I have for now. Please let me know what you think
> > about it.
>
> FWIW virt-manager basically doesn't consume the host topology XML, so
there's
> no concern there.
That's good to know :)
> A quick grep seems to indicate that both nova (openstack) and vdsm
> (ovirt/rhev) _do_ consume this XML for their numa magic (git grep sibling),
> but I can't speak to the details of how it's consumed.
We won't know whether the proposal is actually sensible until
David weighs in, but I'm adding Martin back in the loop so
we can maybe give us the oVirt angle in the meantime.
TBH, I'm not really sure what you want from me. Most of the questions
seem to be libvirt design decisions which are independent of the layers
below.
--
David Gibson <dgibson(a)redhat.com>
Senior Software Engineer, Virtualization, Red Hat
10:52 a.m.
On Tue, 2016-05-31 at 16:08 +1000, David Gibson wrote:
> QEMU fails with errors like
>
> qemu-kvm: Cannot support more than 8 threads on PPC with KVM
> qemu-kvm: Cannot support more than 1 threads on PPC with TCG
>
> depending on the guest type.
Note that in a sense the two errors come about for different reasons.
On Power, to a much greater degree than x86, threads on the same core
have observably different behaviour from threads on different cores.
Because of that, there's no reasonable way for KVM to present more
guest threads-per-core than there are host threads-per-core.
The limit of 1 thread on TCG is simply because no-one's ever bothered
to implement SMT emulation in qemu.
That just means in the future we might have to expose something
other than an hardcoded '1' as guest thread limit for TCG guests;
the interface would remain valid AFAICT.
> physical_core_id would be 32 for all of the above - it would
> just be the very value of core_id the kernel reads from the
> hardware and reports through sysfs.
>
> The tricky bit is that, when subcores are in use, core_id and
> physical_core_id would not match. They will always match on
> architectures that lack the concept of subcores, though.
Yeah, I'm still not terribly convinced that we should even be
presenting physical core info instead of *just* logical core info. If
you care that much about physical core topology, you probably
shouldn't be running your system in subcore mode.
Me neither. We could leave it out initially, and add it later
if it turns out to be useful, I guess.
> > > The optimal guest topology in this case would be
> > >
> > > <vcpu placement='static'
cpuset='4'>4</vcpu>
> > > <cpu>
> > > <topology sockets='1' cores='1'
threads='4'/>
> > > </cpu>
> >
> > So when we pin to logical CPU #4, ppc KVM is smart enough to see that it's
a
> > subcore thread, will then make use of the offline threads in the same subcore?
> > Or does libvirt do anything fancy to facilitate this case?
>
> My understanding is that libvirt shouldn't have to do anything
> to pass the hint to kvm, but David will have the authoritative
> answer here.
Um.. I'm not totally certain. It will be one of two things:
a) you just bind the guest thread to the representative host thread
b) you bind the guest thread to a cpumask with all of the host
threads on the relevant (sub)core - including the offline host
threads
I'll try to figure out which one it is.
I played with this a bit: I created a guest with
<vcpu placement='static' cpuset='0,8'>8</vcpu>
<cpu>
<topology sockets='1' cores='2' threads='4'/>
</cpu>
and then, inside the guest, I used cgroups to pin a bunch
of busy loops to specific vCPUs.
As long as all the load (8+ busy loops) was distributed
only across vCPUs 0-3, one of the host threads remained idle.
As soon as the first of the jobs was moved to vCPUs 4-7, the
other host thread immediately jumped to 100%.
This seems to indicate that QEMU / KVM are actually smart
enough to schedule guest threads on the corresponding host
threads. I think :)
On the other hand, when I changed the guest to distribute the
8 vCPUs among 2 sockets with 4 cores each instead, the second
host thread would start running as soon as I started the
second busy loop.
> We won't know whether the proposal is actually sensible
until
> David weighs in, but I'm adding Martin back in the loop so
> we can maybe give us the oVirt angle in the meantime.
TBH, I'm not really sure what you want from me. Most of the questions
seem to be libvirt design decisions which are independent of the layers
below.
I mostly need you to sanity check my proposals and point out
any incorrect / dubious claims, just like you did above :)
The design of features like this one can have pretty
significant consequences for the interactions between the
various layers, and when the choices are not straightforward
I think it's better to gather as much feedback as possible
from across the stack before moving forward with an
implementation.
--
Andrea Bolognani
Software Engineer - Virtualization Team
6:42 p.m.
On Fri, 10 Jun 2016 17:52:47 +0200
Andrea Bolognani <abologna(a)redhat.com> wrote:
On Tue, 2016-05-31 at 16:08 +1000, David Gibson wrote:
> > QEMU fails with errors like
> >
> > qemu-kvm: Cannot support more than 8 threads on PPC with KVM
> > qemu-kvm: Cannot support more than 1 threads on PPC with TCG
> >
> > depending on the guest type.
>
> Note that in a sense the two errors come about for different reasons.
>
> On Power, to a much greater degree than x86, threads on the same core
> have observably different behaviour from threads on different cores.
> Because of that, there's no reasonable way for KVM to present more
> guest threads-per-core than there are host threads-per-core.
>
> The limit of 1 thread on TCG is simply because no-one's ever bothered
> to implement SMT emulation in qemu.
That just means in the future we might have to expose something
other than an hardcoded '1' as guest thread limit for TCG guests;
the interface would remain valid AFAICT.
Right, that's kind of my point.
> > physical_core_id would be 32 for all of the above - it
would
> > just be the very value of core_id the kernel reads from the
> > hardware and reports through sysfs.
> >
> > The tricky bit is that, when subcores are in use, core_id and
> > physical_core_id would not match. They will always match on
> > architectures that lack the concept of subcores, though.
>
> Yeah, I'm still not terribly convinced that we should even be
> presenting physical core info instead of *just* logical core info. If
> you care that much about physical core topology, you probably
> shouldn't be running your system in subcore mode.
Me neither. We could leave it out initially, and add it later
if it turns out to be useful, I guess.
I think that's a good idea.
> > > > The optimal guest topology in this case would be
> > > >
> > > > <vcpu placement='static'
cpuset='4'>4</vcpu>
> > > > <cpu>
> > > > <topology sockets='1' cores='1'
threads='4'/>
> > > > </cpu>
> > >
> > > So when we pin to logical CPU #4, ppc KVM is smart enough to see that
it's a
> > > subcore thread, will then make use of the offline threads in the same
subcore?
> > > Or does libvirt do anything fancy to facilitate this case?
> >
> > My understanding is that libvirt shouldn't have to do anything
> > to pass the hint to kvm, but David will have the authoritative
> > answer here.
>
> Um.. I'm not totally certain. It will be one of two things:
> a) you just bind the guest thread to the representative host thread
> b) you bind the guest thread to a cpumask with all of the host
> threads on the relevant (sub)core - including the offline host
> threads
>
> I'll try to figure out which one it is.
I played with this a bit: I created a guest with
<vcpu placement='static' cpuset='0,8'>8</vcpu>
<cpu>
<topology sockets='1' cores='2' threads='4'/>
</cpu>
and then, inside the guest, I used cgroups to pin a bunch
of busy loops to specific vCPUs.
As long as all the load (8+ busy loops) was distributed
only across vCPUs 0-3, one of the host threads remained idle.
As soon as the first of the jobs was moved to vCPUs 4-7, the
other host thread immediately jumped to 100%.
This seems to indicate that QEMU / KVM are actually smart
enough to schedule guest threads on the corresponding host
threads. I think :)
Uh.. yes. Guest threads on the same guest core will always be
scheduled together on a physical (sub)core. In fact, it *has* to be
done this way because recent processors contain the msgsnd / msgrcv
instructions which directly send interrupts from one thread to
another. Those instructions are not HV privileged, so they can be
invoked directly by the guest and their behaviour can't be virtualized.
This is one of the ways in which threads on the same core are
guest-observably different from threads on different cores mentioned
above.
On the other hand, when I changed the guest to distribute the
8 vCPUs among 2 sockets with 4 cores each instead, the second
host thread would start running as soon as I started the
second busy loop.
Right likewise a single physical (sub)core can never simultaneously run
threads from multiple guests (or guest and host). msgsnd above, as
well as some other things, would allow one guest to interfere with
another, breaking isolation.
This is the reason that having multiple threads active in the host
while also running guests would be almost impossibly difficult.
> > We won't know whether the proposal is actually sensible
until
> > David weighs in, but I'm adding Martin back in the loop so
> > we can maybe give us the oVirt angle in the meantime.
>
> TBH, I'm not really sure what you want from me. Most of the questions
> seem to be libvirt design decisions which are independent of the layers
> below.
I mostly need you to sanity check my proposals and point out
any incorrect / dubious claims, just like you did above :)
The design of features like this one can have pretty
significant consequences for the interactions between the
various layers, and when the choices are not straightforward
I think it's better to gather as much feedback as possible
from across the stack before moving forward with an
implementation.
--
Andrea Bolognani
Software Engineer - Virtualization Team
--
David Gibson <dgibson(a)redhat.com>
Senior Software Engineer, Virtualization, Red Hat
10:40 a.m.
On Thu, 2016-05-05 at 20:48 +0200, Andrea Bolognani wrote:
On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> The nodeinfo output was fixed earlier to reflect the actual cpus available in
> KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
> look overcommitted when its not. The current fixes are aimed at helping the
> users make better decisions on the kind of guest cpu topology that can be
> supported on the given sucore_per_core setting of KVM host and also hint the
> way to pin the guest vcpus efficiently.
>
> I am planning to add some test cases once the approach is accepted.
>
> With respect to Patch 2:
> The second patch adds a new element to the cpus tag and I need your inputs on
> if that is okay. Also if there is a
> better way. I am not sure if the existing
> clients have RNG checks that might fail with the approach. Or if the checks
> are not enoforced on the elements but only on the tags.
>
> With my approach if the rng checks pass, the new element "capacity" even
if
> ignored by many clients would have no impact except for PPC64.
>
> To the extent I looked at code, the siblings changes dont affect existing
> libvirt functionality. Please do let me know otherwise.
So, I've been going through this old thread trying to figure out
a way to improve the status quo. I'd like to collect as much
feedback as possible, especially from people who have worked in
this area of libvirt before or have written tools based on it.
I forgot to link this OpenStack Nova spec[1] that
Shivaprasad pointed me to earlier.
[1] https://review.openstack.org/gitweb?p=openstack/nova-specs.git;a=commi...
c5
--
Andrea Bolognani
Software Engineer - Virtualization Team
9:35 a.m.
On Thu, May 05, 2016 at 08:48:05PM +0200, Andrea Bolognani wrote:
On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> The nodeinfo output was fixed earlier to reflect the actual cpus available in
> KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
> look overcommitted when its not. The current fixes are aimed at helping the
> users make better decisions on the kind of guest cpu topology that can be
> supported on the given sucore_per_core setting of KVM host and also hint the
> way to pin the guest vcpus efficiently.
>
> I am planning to add some test cases once the approach is accepted.
>
> With respect to Patch 2:
> The second patch adds a new element to the cpus tag and I need your inputs on
> if that is okay. Also if there is a better way. I am not sure if the existing
> clients have RNG checks that might fail with the approach. Or if the checks
> are not enoforced on the elements but only on the tags.
>
> With my approach if the rng checks pass, the new element "capacity" even
if
> ignored by many clients would have no impact except for PPC64.
>
> To the extent I looked at code, the siblings changes dont affect existing
> libvirt functionality. Please do let me know otherwise.
So, I've been going through this old thread trying to figure out
a way to improve the status quo. I'd like to collect as much
feedback as possible, especially from people who have worked in
this area of libvirt before or have written tools based on it.
As hinted above, this series is really trying to address two
different issue, and I think it's helpful to reason about them
separately.
** Guest threads limit **
My dual-core laptop will happily run a guest configured with
<cpu>
<topology sockets='1' cores='1' threads='128'/>
</cpu>
but POWER guests are limited to 8/subcores_per_core threads.
We need to report this information to the user somehow, and
I can't see an existing place where it would fit nicely. We
definitely don't want to overload the meaning of an existing
element/attribute with this. It should also only appear in
the (dom)capabilities XML of ppc64 hosts.
I don't think this is too problematic or controversial, we
just need to pick a nice place to display this information.
** Efficient guest topology **
To achieve optimal performance, you want to match guest
threads with host threads.
On x86, you can choose suitable host threads by looking at
the capabilities XML: the presence of elements like
<cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
<cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
means you should configure your guest to use
<vcpu placement='static' cpuset='2-3'>2</vcpu>
<cpu>
<topology sockets='1' cores='1' threads='2'/>
</cpu>
Notice how siblings can be found either looking at the
attribute with the same name, or by matching them using the
value of the core_id attribute. Also notice how you are
supposed to pin as many vCPUs as the number of elements in
the cpuset - one guest thread per host thread.
On POWER, this gets much trickier: only the *primary* thread
of each (sub)core appears to be online in the host, but all
threads can actually have a vCPU running on them. So
<cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
<cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
which is what you'd get with subcores_per_core=2, is very
confusing.
The optimal guest topology in this case would be
<vcpu placement='static' cpuset='4'>4</vcpu>
<cpu>
<topology sockets='1' cores='1' threads='4'/>
</cpu>
but neither approaches mentioned above work to figure out the
correct value for the cpuset attribute.
In this case, a possible solution would be to alter the values
of the core_id and siblings attribute such that both would be
the same as the id attribute, which would naturally make both
approaches described above work.
Additionaly, a new attribute would be introduced to serve as
a multiplier for the "one guest thread per host thread" rule
mentioned earlier: the resulting XML would look like
<cpu id='0' socket_id='0' core_id='0' siblings='0'
capacity='4'/>
<cpu id='4' socket_id='0' core_id='4' siblings='4'
capacity='4'/>
which contains all the information needed to build the right
guest topology. The capacity attribute would have value 1 on
all architectures except for ppc64.
I don't really like the fact that with this design, we effectively
have a bunch of <cpu> which are invisible whose existance is just
implied by the 'capacity=4' attribute.
I also don't like tailoring output of capabilities XML for one
specific use case.
IOW, I think we should explicitly represent all the CPUs in the
node capabilities, even if they are offline in the host. We could
introduce a new attribute to indicate the status of CPUs. So
instead of
<cpu id='0' socket_id='0' core_id='0' siblings='0'
capacity='4'/>
<cpu id='4' socket_id='0' core_id='4' siblings='4'
capacity='4'/>
I'd like to see
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="online"/>
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="online"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
The domain capabilities meanwhile is where you'd express any usage
constraint for cores/threads requried by QEMU.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
7:33 a.m.
On Tue, 2016-07-19 at 15:35 +0100, Daniel P. Berrange wrote:
On Thu, May 05, 2016 at 08:48:05PM +0200, Andrea Bolognani wrote:
>
> On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> >
> > The nodeinfo output was fixed earlier to reflect the actual cpus available in
> > KVM mode on PPC64. The earlier fixes covered the aspect of not making a host
> > look overcommitted when its not. The current fixes are aimed at helping the
> > users make better decisions on the kind of guest cpu topology that can be
> > supported on the given sucore_per_core setting of KVM host and also hint the
> > way to pin the guest vcpus efficiently.
> >
> > I am planning to add some test cases once the approach is accepted.
> >
> > With respect to Patch 2:
> > The second patch adds a new element to the cpus tag and I need your inputs on
> > if that is okay. Also if there is a better way. I am not sure if the existing
> > clients have RNG checks that might fail with the approach. Or if the checks
> > are not enoforced on the elements but only on the tags.
> >
> > With my approach if the rng checks pass, the new element "capacity"
even if
> > ignored by many clients would have no impact except for PPC64.
> >
> > To the extent I looked at code, the siblings changes dont affect existing
> > libvirt functionality. Please do let me know otherwise.
>
> So, I've been going through this old thread trying to figure out
> a way to improve the status quo. I'd like to collect as much
> feedback as possible, especially from people who have worked in
> this area of libvirt before or have written tools based on it.
>
> As hinted above, this series is really trying to address two
> different issue, and I think it's helpful to reason about them
> separately.
>
>
> ** Guest threads limit **
>
> My dual-core laptop will happily run a guest configured with
>
> <cpu>
> <topology sockets='1' cores='1' threads='128'/>
> </cpu>
>
> but POWER guests are limited to 8/subcores_per_core threads.
>
> We need to report this information to the user somehow, and
> I can't see an existing place where it would fit nicely. We
> definitely don't want to overload the meaning of an existing
> element/attribute with this. It should also only appear in
> the (dom)capabilities XML of ppc64 hosts.
>
> I don't think this is too problematic or controversial, we
> just need to pick a nice place to display this information.
>
>
> ** Efficient guest topology **
>
> To achieve optimal performance, you want to match guest
> threads with host threads.
>
> On x86, you can choose suitable host threads by looking at
> the capabilities XML: the presence of elements like
>
> <cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
> <cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
>
> means you should configure your guest to use
>
> <vcpu placement='static' cpuset='2-3'>2</vcpu>
> <cpu>
> <topology sockets='1' cores='1' threads='2'/>
> </cpu>
>
> Notice how siblings can be found either looking at the
> attribute with the same name, or by matching them using the
> value of the core_id attribute. Also notice how you are
> supposed to pin as many vCPUs as the number of elements in
> the cpuset - one guest thread per host thread.
>
> On POWER, this gets much trickier: only the *primary* thread
> of each (sub)core appears to be online in the host, but all
> threads can actually have a vCPU running on them. So
>
> <cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
> <cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
>
> which is what you'd get with subcores_per_core=2, is very
> confusing.
>
> The optimal guest topology in this case would be
>
> <vcpu placement='static' cpuset='4'>4</vcpu>
> <cpu>
> <topology sockets='1' cores='1' threads='4'/>
> </cpu>
>
> but neither approaches mentioned above work to figure out the
> correct value for the cpuset attribute.
>
> In this case, a possible solution would be to alter the values
> of the core_id and siblings attribute such that both would be
> the same as the id attribute, which would naturally make both
> approaches described above work.
>
> Additionaly, a new attribute would be introduced to serve as
> a multiplier for the "one guest thread per host thread" rule
> mentioned earlier: the resulting XML would look like
>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0' capacity='4'/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4' capacity='4'/>
>
> which contains all the information needed to build the right
> guest topology. The capacity attribute would have value 1 on
> all architectures except for ppc64.
I don't really like the fact that with this design, we effectively
have a bunch of <cpu> which are invisible whose existance is just
implied by the 'capacity=4' attribute.
I also don't like tailoring output of capabilities XML for one
specific use case.
IOW, I think we should explicitly represent all the CPUs in the
node capabilities, even if they are offline in the host. We could
introduce a new attribute to indicate the status of CPUs. So
instead of
<cpu id='0' socket_id='0' core_id='0' siblings='0'
capacity='4'/>
<cpu id='4' socket_id='0' core_id='4' siblings='4'
capacity='4'/>
I'd like to see
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="online"/>
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="online"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
I assume you meant
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="online"/>
<cpu id='1' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='2' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='3' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="online"/>
<cpu id='5' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='6' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='7' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
and that this would represent a configuration where
subcores-per-core=2, eg. CPUs 0-7 belong to the same physical
core but to different logical cores (subcores).
IIRC doing something like this was proposed at some point in
the past, but was rejected on the ground that existing tools
assume that 1) CPUs listed in the NUMA topology are online
and 2) you can pin vCPUs to them. So they would try to pin
vCPUs to eg. CPU 1 and that will fail.
Additionally, this doesn't tell us anything about whether any
host CPU can run a guest CPU: given the above configuration,
on ppc64, we know that CPU 1 can run guest threads even though
it's offline because CPU 0 is online, but the same isn't true
on x86.
So we would end up needing three new boolean properties:
- online - whether the CPU is online
- can_run_vcpus - whether the CPU can run vCPUs
- can_pin_vcpus - whether vCPUs can be pinned to the CPU
and all higher level tools would have to adapt to use them.
Existing logic would not work with newer libvirt versions,
and x86 would be affected by these changes as well.
One more thing: since the kernel doesn't expose information
about offline CPUs, we'll have to figure out those ourselves:
we're already doing that, to some degree, on ppc64, but there
are some cases where it's just impossible to do so reliably.
When that happens, we throw our hands up in the air and return
a completely bogus topology. That would suddenly be the case
on x86 as well.
So yeah... Tricky stuff. But thanks for providing some input,
and please keep it coming! :)
The domain capabilities meanwhile is where you'd express any
usage
constraint for cores/threads requried by QEMU.
Regards,
Daniel
--
Andrea Bolognani / Red Hat / Virtualization
7:49 a.m.
On Wed, Jul 20, 2016 at 02:33:33PM +0200, Andrea Bolognani wrote:
On Tue, 2016-07-19 at 15:35 +0100, Daniel P. Berrange wrote:
> On Thu, May 05, 2016 at 08:48:05PM +0200, Andrea Bolognani wrote:
> >
> > On Fri, 2016-01-29 at 01:32 -0500, Shivaprasad G Bhat wrote:
> > >
> > > The nodeinfo output was fixed earlier to reflect the actual cpus available
in
> > > KVM mode on PPC64. The earlier fixes covered the aspect of not making a
host
> > > look overcommitted when its not. The current fixes are aimed at helping
the
> > > users make better decisions on the kind of guest cpu topology that can be
> > > supported on the given sucore_per_core setting of KVM host and also hint
the
> > > way to pin the guest vcpus efficiently.
> > >
> > > I am planning to add some test cases once the approach is accepted.
> > >
> > > With respect to Patch 2:
> > > The second patch adds a new element to the cpus tag and I need your inputs
on
> > > if that is okay. Also if there is a better way. I am not sure if the
existing
> > > clients have RNG checks that might fail with the approach. Or if the
checks
> > > are not enoforced on the elements but only on the tags.
> > >
> > > With my approach if the rng checks pass, the new element
"capacity" even if
> > > ignored by many clients would have no impact except for PPC64.
> > >
> > > To the extent I looked at code, the siblings changes dont affect existing
> > > libvirt functionality. Please do let me know otherwise.
> >
> > So, I've been going through this old thread trying to figure out
> > a way to improve the status quo. I'd like to collect as much
> > feedback as possible, especially from people who have worked in
> > this area of libvirt before or have written tools based on it.
> >
> > As hinted above, this series is really trying to address two
> > different issue, and I think it's helpful to reason about them
> > separately.
> >
> >
> > ** Guest threads limit **
> >
> > My dual-core laptop will happily run a guest configured with
> >
> > <cpu>
> > <topology sockets='1' cores='1'
threads='128'/>
> > </cpu>
> >
> > but POWER guests are limited to 8/subcores_per_core threads.
> >
> > We need to report this information to the user somehow, and
> > I can't see an existing place where it would fit nicely. We
> > definitely don't want to overload the meaning of an existing
> > element/attribute with this. It should also only appear in
> > the (dom)capabilities XML of ppc64 hosts.
> >
> > I don't think this is too problematic or controversial, we
> > just need to pick a nice place to display this information.
> >
> >
> > ** Efficient guest topology **
> >
> > To achieve optimal performance, you want to match guest
> > threads with host threads.
> >
> > On x86, you can choose suitable host threads by looking at
> > the capabilities XML: the presence of elements like
> >
> > <cpu id='2' socket_id='0' core_id='1'
siblings='2-3'/>
> > <cpu id='3' socket_id='0' core_id='1'
siblings='2-3'/>
> >
> > means you should configure your guest to use
> >
> > <vcpu placement='static' cpuset='2-3'>2</vcpu>
> > <cpu>
> > <topology sockets='1' cores='1'
threads='2'/>
> > </cpu>
> >
> > Notice how siblings can be found either looking at the
> > attribute with the same name, or by matching them using the
> > value of the core_id attribute. Also notice how you are
> > supposed to pin as many vCPUs as the number of elements in
> > the cpuset - one guest thread per host thread.
> >
> > On POWER, this gets much trickier: only the *primary* thread
> > of each (sub)core appears to be online in the host, but all
> > threads can actually have a vCPU running on them. So
> >
> > <cpu id='0' socket_id='0' core_id='32'
siblings='0,4'/>
> > <cpu id='4' socket_id='0' core_id='32'
siblings='0,4'/>
> >
> > which is what you'd get with subcores_per_core=2, is very
> > confusing.
> >
> > The optimal guest topology in this case would be
> >
> > <vcpu placement='static' cpuset='4'>4</vcpu>
> > <cpu>
> > <topology sockets='1' cores='1'
threads='4'/>
> > </cpu>
> >
> > but neither approaches mentioned above work to figure out the
> > correct value for the cpuset attribute.
> >
> > In this case, a possible solution would be to alter the values
> > of the core_id and siblings attribute such that both would be
> > the same as the id attribute, which would naturally make both
> > approaches described above work.
> >
> > Additionaly, a new attribute would be introduced to serve as
> > a multiplier for the "one guest thread per host thread" rule
> > mentioned earlier: the resulting XML would look like
> >
> > <cpu id='0' socket_id='0' core_id='0'
siblings='0' capacity='4'/>
> > <cpu id='4' socket_id='0' core_id='4'
siblings='4' capacity='4'/>
> >
> > which contains all the information needed to build the right
> > guest topology. The capacity attribute would have value 1 on
> > all architectures except for ppc64.
>
> I don't really like the fact that with this design, we effectively
> have a bunch of <cpu> which are invisible whose existance is just
> implied by the 'capacity=4' attribute.
>
> I also don't like tailoring output of capabilities XML for one
> specific use case.
>
> IOW, I think we should explicitly represent all the CPUs in the
> node capabilities, even if they are offline in the host. We could
> introduce a new attribute to indicate the status of CPUs. So
> instead of
>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0' capacity='4'/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4' capacity='4'/>
>
> I'd like to see
>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0-3' state="online"/>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0-3' state="offline"/>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0-3' state="offline"/>
> <cpu id='0' socket_id='0' core_id='0'
siblings='0-3' state="offline"/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4-7' state="online"/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4-7' state="offline"/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4-7' state="offline"/>
> <cpu id='4' socket_id='0' core_id='4'
siblings='4-7' state="offline"/>
I assume you meant
<cpu id='0' socket_id='0' core_id='0' siblings='0-3'
state="online"/>
<cpu id='1' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='2' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='3' socket_id='0' core_id='0' siblings='0-3'
state="offline"/>
<cpu id='4' socket_id='0' core_id='4' siblings='4-7'
state="online"/>
<cpu id='5' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='6' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
<cpu id='7' socket_id='0' core_id='4' siblings='4-7'
state="offline"/>
Sigh, yes, of course it needs unique 'id' values :-)
and that this would represent a configuration where
subcores-per-core=2, eg. CPUs 0-7 belong to the same physical
core but to different logical cores (subcores).
IIRC doing something like this was proposed at some point in
the past, but was rejected on the ground that existing tools
assume that 1) CPUs listed in the NUMA topology are online
and 2) you can pin vCPUs to them. So they would try to pin
vCPUs to eg. CPU 1 and that will fail.
We could easily deal with this by adding a flag to the
virConnectGetCapabilities() API eg VIR_CONNECT_CAPS_ALL_CPUS
so we only show the expanded full CPU list if an app opts
in to it.
Additionally, this doesn't tell us anything about whether any
host CPU can run a guest CPU: given the above configuration,
on ppc64, we know that CPU 1 can run guest threads even though
it's offline because CPU 0 is online, but the same isn't true
on x86.
So we would end up needing three new boolean properties:
- online - whether the CPU is online
- can_run_vcpus - whether the CPU can run vCPUs
- can_pin_vcpus - whether vCPUs can be pinned to the CPU
These two can*vcpus props aren't telling the whole story
because they are assuming use of KVM - TCG or LXC guests
would not follow the same rules for runability here.
This is why I think the host capabilities XML should focus
on describing what hardware actually exists and its state,
and not say anything about guest runnability.
Historically we've let apps figure out the runnability
of pCPUs vs guest vCPUs, but even on x86 it isn't as simple
as you/we make it out to be.
For example, nothing here reflects the fact that the host
OS could have /sys/fs/cgroups/cpuset/cpuset.cpus configured
to a subset of CPUs. So already today on x86, just because
a CPU is listed in the capabilities XML, does not imply that
you can run a guest on it.
So I think there's a gap for exposing information about
runability of guests vs host CPUs, that does not really
fit in the host capabilities. Possibly it could be in the
guest capabilities, but more likely it would need to be
a new API entirely.
and all higher level tools would have to adapt to use them.
Existing logic would not work with newer libvirt versions,
and x86 would be affected by these changes as well.
Yes, as mentioned above, apps would have to opt-in to using
the new information.
One more thing: since the kernel doesn't expose information
about offline CPUs, we'll have to figure out those ourselves:
we're already doing that, to some degree, on ppc64, but there
are some cases where it's just impossible to do so reliably.
When that happens, we throw our hands up in the air and return
a completely bogus topology. That would suddenly be the case
on x86 as well.
So yeah... Tricky stuff. But thanks for providing some input,
and please keep it coming! :)
> The domain capabilities meanwhile is where you'd express any usage
> constraint for cores/threads requried by QEMU.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
10:52 a.m.
On Wed, 2016-07-20 at 13:49 +0100, Daniel P. Berrange wrote:
> IIRC doing something like this was proposed at some point in
> the past, but was rejected on the ground that existing tools
> assume that 1) CPUs listed in the NUMA topology are online
> and 2) you can pin vCPUs to them. So they would try to pin
> vCPUs to eg. CPU 1 and that will fail.
We could easily deal with this by adding a flag to the
virConnectGetCapabilities() API eg VIR_CONNECT_CAPS_ALL_CPUS
so we only show the expanded full CPU list if an app opts
in to it.
Okay.
> Additionally, this doesn't tell us anything about whether
any
> host CPU can run a guest CPU: given the above configuration,
> on ppc64, we know that CPU 1 can run guest threads even though
> it's offline because CPU 0 is online, but the same isn't true
> on x86.
>
> So we would end up needing three new boolean properties:
>
> - online - whether the CPU is online
> - can_run_vcpus - whether the CPU can run vCPUs
> - can_pin_vcpus - whether vCPUs can be pinned to the CPU
These two can*vcpus props aren't telling the whole story
because they are assuming use of KVM - TCG or LXC guests
would not follow the same rules for runability here.
This is why I think the host capabilities XML should focus
on describing what hardware actually exists and its state,
and not say anything about guest runnability.
Historically we've let apps figure out the runnability
of pCPUs vs guest vCPUs, but even on x86 it isn't as simple
as you/we make it out to be.
For example, nothing here reflects the fact that the host
OS could have /sys/fs/cgroups/cpuset/cpuset.cpus configured
to a subset of CPUs. So already today on x86, just because
a CPU is listed in the capabilities XML, does not imply that
you can run a guest on it.
So I think there's a gap for exposing information about
runability of guests vs host CPUs, that does not really
fit in the host capabilities. Possibly it could be in the
guest capabilities, but more likely it would need to be
a new API entirely.
Why wouldn't domcapabilities be a good place? We already
have some related information (the <vcpu> tag).
In any case, regardless of whether or not it will ultimately
be part of domcapabilities, I guess a good starting point is
to sketch out how the xml would look like. I'm thinking of
something like
<cpu id='0' runnable='yes' pinnable='yes'
run_group='0-3'/>
<cpu id='1' runnable='yes' pinnable='no'
run_group='0-3'/>
<cpu id='2' runnable='yes' pinnable='no'
run_group='0-3'/>
<cpu id='3' runnable='yes' pinnable='no'
run_group='0-3'/>
where 'runnable' tells you whether the CPU can run vCPUs,
'pinnable' whether vCPUs can be pinned, and 'run_group' tells
you what CPUs the pinned vCPUs will actually run on? On x86
it would look simpler:
<cpu id='0' runnable='yes' pinnable='yes'
run_group='0'/>
<cpu id='1' runnable='yes' pinnable='yes'
run_group='1'/>
I think we don't need to add information that can already be
obtained from existing capabilities, such as the siblings
list.
The value of 'runnable' etc. would be calculated taking into
account the architecture, the hypervisor, and host
configuration details such as the ones you mentioned above.
Does the above look reasonable? Anything I'm missing?
--
Andrea Bolognani / Red Hat / Virtualization
11:18 a.m.
On Thu, Jul 21, 2016 at 05:52:11PM +0200, Andrea Bolognani wrote:
On Wed, 2016-07-20 at 13:49 +0100, Daniel P. Berrange wrote:
> > Additionally, this doesn't tell us anything about whether any
> > host CPU can run a guest CPU: given the above configuration,
> > on ppc64, we know that CPU 1 can run guest threads even though
> > it's offline because CPU 0 is online, but the same isn't true
> > on x86.
> >
> > So we would end up needing three new boolean properties:
> >
> > - online - whether the CPU is online
> > - can_run_vcpus - whether the CPU can run vCPUs
> > - can_pin_vcpus - whether vCPUs can be pinned to the CPU
>
> These two can*vcpus props aren't telling the whole story
> because they are assuming use of KVM - TCG or LXC guests
> would not follow the same rules for runability here.
>
> This is why I think the host capabilities XML should focus
> on describing what hardware actually exists and its state,
> and not say anything about guest runnability.
>
> Historically we've let apps figure out the runnability
> of pCPUs vs guest vCPUs, but even on x86 it isn't as simple
> as you/we make it out to be.
>
> For example, nothing here reflects the fact that the host
> OS could have /sys/fs/cgroups/cpuset/cpuset.cpus configured
> to a subset of CPUs. So already today on x86, just because
> a CPU is listed in the capabilities XML, does not imply that
> you can run a guest on it.
>
> So I think there's a gap for exposing information about
> runability of guests vs host CPUs, that does not really
> fit in the host capabilities. Possibly it could be in the
> guest capabilities, but more likely it would need to be
> a new API entirely.
Why wouldn't domcapabilities be a good place? We already
have some related information (the <vcpu> tag).
Just depends whether we have all the info we need available
in the domcapabilities API. eg I was wondering whether we
would need info about the guest CPU topology (sockets, core,
threads) too. If we don't, then great, it can be in the
domcapabilities.
In any case, regardless of whether or not it will ultimately
be part of domcapabilities, I guess a good starting point is
to sketch out how the xml would look like. I'm thinking of
something like
<cpu id='0' runnable='yes' pinnable='yes'
run_group='0-3'/>
<cpu id='1' runnable='yes' pinnable='no'
run_group='0-3'/>
<cpu id='2' runnable='yes' pinnable='no'
run_group='0-3'/>
<cpu id='3' runnable='yes' pinnable='no'
run_group='0-3'/>
where 'runnable' tells you whether the CPU can run vCPUs,
'pinnable' whether vCPUs can be pinned, and 'run_group' tells
you what CPUs the pinned vCPUs will actually run on? On x86
What's the relationship to guest CPUs and their topology
here ? Is this trying to say that all vCPUs placed in a
run_group must be in the same virtual socket ?
If so is the pinnable attribute trying to imply that when
you change pinning for a vCPU on the first pCPU in a run
group, that it will automatically change pinning of the
other vCPUs on that same pCPU run group ?
it would look simpler:
<cpu id='0' runnable='yes' pinnable='yes'
run_group='0'/>
<cpu id='1' runnable='yes' pinnable='yes'
run_group='1'/>
I think we don't need to add information that can already be
obtained from existing capabilities, such as the siblings
list.
Yep, it'd be nice to avoid duplicating info already exposed in
the host capabilities, such as host topology.
It feels like 'run_group' is however rather duplicating the
info. eg, isn't 'run_group' just directly saying which
cores are part of the same socket.
For sake of clarity can you just back up again & explain exactly
what the rules are wrt PPC & pCPU / vCPU topology and plaement.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
3076
days inactive
3250
days old
17 comments
5 participants
participants (5)
-
Andrea Bolognani -
Cole Robinson -
Daniel P. Berrange -
David Gibson -
Shivaprasad G Bhat