[PATCH] docs: Improve documentation for dies and clusters
I've seen examples in the wild of the cluster attribute having non-zero value on x86_64. This is obviously quite confusing, but it's the information that Linux exposes to userspace and we don't really have a way to tell apart a valid die/cluster ID from a dummy one. What ultimately matters is that the underlying assumptions about topology are respected, which they are: in the x86_64 cases that I have analyzed, for example, each "cluster" contained exactly one core, so any program that would use this information to influence guest topology decisions would be unaffected by the additional level showing up in the hierarchy. In an attempt to reduce confusion, document that the value for these attributes is not necessarily going to be zero. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- docs/formatcaps.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/formatcaps.rst b/docs/formatcaps.rst index f37532296f..c15d391b63 100644 --- a/docs/formatcaps.rst +++ b/docs/formatcaps.rst @@ -74,14 +74,18 @@ The ``<host/>`` element consists of the following child elements: ``die_id`` Identifier for the die the CPU is in. - Note that not all architectures support CPU dies: if the current - architecture doesn't, the value will be 0 for all CPUs. + Note that, while not all architectures support CPU dies, this attribute + will always be present in the capabilities XML. If the architecture + doesn't support them, the value will likely be 0 for all CPUs, but it + could also be some other arbitrary value. ``cluster_id`` Identifier for the cluster the CPU is in. - Note that not all architectures support CPU clusters: if the current - architecture doesn't, the value will be 0 for all CPUs. + Note that, while not all architectures support CPU clusters, this + attribute will always be present in the capabilities XML. If the + architecture doesn't support them, the value will likely be 0 for all + CPUs, but it could also be some other arbitrary value. ``core_id`` Identifier for the core the CPU is in. -- 2.43.0
On Tue, Feb 06, 2024 at 07:04:30PM +0100, Andrea Bolognani wrote:
I've seen examples in the wild of the cluster attribute having non-zero value on x86_64.
This is obviously quite confusing, but it's the information that Linux exposes to userspace and we don't really have a way to tell apart a valid die/cluster ID from a dummy one.
What ultimately matters is that the underlying assumptions about topology are respected, which they are: in the x86_64 cases that I have analyzed, for example, each "cluster" contained exactly one core, so any program that would use this information to influence guest topology decisions would be unaffected by the additional level showing up in the hierarchy.
In an attempt to reduce confusion, document that the value for these attributes is not necessarily going to be zero.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- docs/formatcaps.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/docs/formatcaps.rst b/docs/formatcaps.rst index f37532296f..c15d391b63 100644 --- a/docs/formatcaps.rst +++ b/docs/formatcaps.rst @@ -74,14 +74,18 @@ The ``<host/>`` element consists of the following child elements: ``die_id`` Identifier for the die the CPU is in.
- Note that not all architectures support CPU dies: if the current - architecture doesn't, the value will be 0 for all CPUs. + Note that, while not all architectures support CPU dies, this attribute + will always be present in the capabilities XML. If the architecture + doesn't support them, the value will likely be 0 for all CPUs, but it + could also be some other arbitrary value.
I would remove the caveat about "not all architectures support CPU dies" entirely, and about special die id values. I tend to say that every CPU every manufactured supports "dies", but that most CPUs dont support more than 1 die :-) The purpose of exposing this info is primarily to help apps / admins in placing guests, and matching host / guest topology where applicable. To do that they don't need to care if a CPU supports more than 1 die or not, they just need to see the topology reported. If they do want to detect >1 die for some reason though, they should not try to look for special 'die_id' values, instead look to see if there are more than 1 *distinct* die_id within a single socket_id.
``cluster_id`` Identifier for the cluster the CPU is in.
- Note that not all architectures support CPU clusters: if the current - architecture doesn't, the value will be 0 for all CPUs. + Note that, while not all architectures support CPU clusters, this + attribute will always be present in the capabilities XML. If the + architecture doesn't support them, the value will likely be 0 for all + CPUs, but it could also be some other arbitrary value.
Same comment as above.
``core_id`` Identifier for the core the CPU is in. -- 2.43.0 _______________________________________________ Devel mailing list -- devel@lists.libvirt.org To unsubscribe send an email to devel-leave@lists.libvirt.org
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Feb 06, 2024 at 06:42:22PM +0000, Daniel P. Berrangé wrote:
On Tue, Feb 06, 2024 at 07:04:30PM +0100, Andrea Bolognani wrote:
``die_id`` Identifier for the die the CPU is in.
+ Note that, while not all architectures support CPU dies, this attribute + will always be present in the capabilities XML. If the architecture + doesn't support them, the value will likely be 0 for all CPUs, but it + could also be some other arbitrary value.
I would remove the caveat about "not all architectures support CPU dies" entirely, and about special die id values. I tend to say that every CPU every manufactured supports "dies", but that most CPUs dont support more than 1 die :-)
The purpose of exposing this info is primarily to help apps / admins in placing guests, and matching host / guest topology where applicable. To do that they don't need to care if a CPU supports more than 1 die or not, they just need to see the topology reported.
If they do want to detect >1 die for some reason though, they should not try to look for special 'die_id' values, instead look to see if there are more than 1 *distinct* die_id within a single socket_id.
That's fair, but I think it could still be useful to highlight that the presence of the attribute in the *host* topology doesn't indicate that attempting to use it in the *guest* topology will work, just to avoid confusion. How about Note that this attribute is always present, even when the architecture doesn't support guests with multiple CPU dies. (and the same for clusters, of course)? -- Andrea Bolognani / Red Hat / Virtualization
On Thu, Feb 08, 2024 at 07:37:00AM -0800, Andrea Bolognani wrote:
On Tue, Feb 06, 2024 at 06:42:22PM +0000, Daniel P. Berrangé wrote:
On Tue, Feb 06, 2024 at 07:04:30PM +0100, Andrea Bolognani wrote:
``die_id`` Identifier for the die the CPU is in.
+ Note that, while not all architectures support CPU dies, this attribute + will always be present in the capabilities XML. If the architecture + doesn't support them, the value will likely be 0 for all CPUs, but it + could also be some other arbitrary value.
I would remove the caveat about "not all architectures support CPU dies" entirely, and about special die id values. I tend to say that every CPU every manufactured supports "dies", but that most CPUs dont support more than 1 die :-)
The purpose of exposing this info is primarily to help apps / admins in placing guests, and matching host / guest topology where applicable. To do that they don't need to care if a CPU supports more than 1 die or not, they just need to see the topology reported.
If they do want to detect >1 die for some reason though, they should not try to look for special 'die_id' values, instead look to see if there are more than 1 *distinct* die_id within a single socket_id.
That's fair, but I think it could still be useful to highlight that the presence of the attribute in the *host* topology doesn't indicate that attempting to use it in the *guest* topology will work, just to avoid confusion.
How about
Note that this attribute is always present, even when the architecture doesn't support guests with multiple CPU dies.
The host topology info says nothing about what's possible in the guest. The only requirement for the guest is that QEMU is new enough to configure this. Aside from that any combo of sockets/ clusters/dies/cores/threads can be configured for the guest, regardless of what host topology reports. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Feb 08, 2024 at 03:47:06PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 07:37:00AM -0800, Andrea Bolognani wrote:
On Tue, Feb 06, 2024 at 06:42:22PM +0000, Daniel P. Berrangé wrote:
On Tue, Feb 06, 2024 at 07:04:30PM +0100, Andrea Bolognani wrote:
``die_id`` Identifier for the die the CPU is in.
+ Note that, while not all architectures support CPU dies, this attribute + will always be present in the capabilities XML. If the architecture + doesn't support them, the value will likely be 0 for all CPUs, but it + could also be some other arbitrary value.
I would remove the caveat about "not all architectures support CPU dies" entirely, and about special die id values. I tend to say that every CPU every manufactured supports "dies", but that most CPUs dont support more than 1 die :-)
The purpose of exposing this info is primarily to help apps / admins in placing guests, and matching host / guest topology where applicable. To do that they don't need to care if a CPU supports more than 1 die or not, they just need to see the topology reported.
If they do want to detect >1 die for some reason though, they should not try to look for special 'die_id' values, instead look to see if there are more than 1 *distinct* die_id within a single socket_id.
That's fair, but I think it could still be useful to highlight that the presence of the attribute in the *host* topology doesn't indicate that attempting to use it in the *guest* topology will work, just to avoid confusion.
How about
Note that this attribute is always present, even when the architecture doesn't support guests with multiple CPU dies.
The host topology info says nothing about what's possible in the guest. The only requirement for the guest is that QEMU is new enough to configure this. Aside from that any combo of sockets/ clusters/dies/cores/threads can be configured for the guest, regardless of what host topology reports.
That's not correct. Even the latest version of QEMU only allows configuring multiple dies and clusters for certain architectures. $ qemu-system-x86_64 -M q35 -smp clusters=2 qemu-system-x86_64: clusters not supported by this machine's CPU topology $ qemu-system-aarch64 -M virt -smp dies=2 qemu-system-aarch64: dies not supported by this machine's CPU topology -- Andrea Bolognani / Red Hat / Virtualization
On Thu, Feb 08, 2024 at 07:52:00AM -0800, Andrea Bolognani wrote:
On Thu, Feb 08, 2024 at 03:47:06PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 07:37:00AM -0800, Andrea Bolognani wrote:
On Tue, Feb 06, 2024 at 06:42:22PM +0000, Daniel P. Berrangé wrote:
On Tue, Feb 06, 2024 at 07:04:30PM +0100, Andrea Bolognani wrote:
``die_id`` Identifier for the die the CPU is in.
+ Note that, while not all architectures support CPU dies, this attribute + will always be present in the capabilities XML. If the architecture + doesn't support them, the value will likely be 0 for all CPUs, but it + could also be some other arbitrary value.
I would remove the caveat about "not all architectures support CPU dies" entirely, and about special die id values. I tend to say that every CPU every manufactured supports "dies", but that most CPUs dont support more than 1 die :-)
The purpose of exposing this info is primarily to help apps / admins in placing guests, and matching host / guest topology where applicable. To do that they don't need to care if a CPU supports more than 1 die or not, they just need to see the topology reported.
If they do want to detect >1 die for some reason though, they should not try to look for special 'die_id' values, instead look to see if there are more than 1 *distinct* die_id within a single socket_id.
That's fair, but I think it could still be useful to highlight that the presence of the attribute in the *host* topology doesn't indicate that attempting to use it in the *guest* topology will work, just to avoid confusion.
How about
Note that this attribute is always present, even when the architecture doesn't support guests with multiple CPU dies.
The host topology info says nothing about what's possible in the guest. The only requirement for the guest is that QEMU is new enough to configure this. Aside from that any combo of sockets/ clusters/dies/cores/threads can be configured for the guest, regardless of what host topology reports.
That's not correct. Even the latest version of QEMU only allows configuring multiple dies and clusters for certain architectures.
Yes but that's a constraint of the guest machine type choice, related to the host capabilities info. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Feb 08, 2024 at 03:54:20PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 07:52:00AM -0800, Andrea Bolognani wrote:
That's fair, but I think it could still be useful to highlight that the presence of the attribute in the *host* topology doesn't indicate that attempting to use it in the *guest* topology will work, just to avoid confusion.
How about
Note that this attribute is always present, even when the architecture doesn't support guests with multiple CPU dies.
The host topology info says nothing about what's possible in the guest. The only requirement for the guest is that QEMU is new enough to configure this. Aside from that any combo of sockets/ clusters/dies/cores/threads can be configured for the guest, regardless of what host topology reports.
That's not correct. Even the latest version of QEMU only allows configuring multiple dies and clusters for certain architectures.
Yes but that's a constraint of the guest machine type choice, related to the host capabilities info.
AFAICT there's no x86_64 (or ppc64, or s390x, or ...) machine type that can use CPU clusters. So if you're planning to run a same-architecture VM (most common use case) you are not going to be able to use CPU clusters, which might be confusing considering that CPU clusters appear to be supported by your host CPU according to the capabilities XML. This is the potential source of confusion that I'm trying to address with the patch. I don't see how *not* mentioning this is going to be more helpful to users. -- Andrea Bolognani / Red Hat / Virtualization
On Thu, Feb 08, 2024 at 08:13:26AM -0800, Andrea Bolognani wrote:
On Thu, Feb 08, 2024 at 03:54:20PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 07:52:00AM -0800, Andrea Bolognani wrote:
That's fair, but I think it could still be useful to highlight that the presence of the attribute in the *host* topology doesn't indicate that attempting to use it in the *guest* topology will work, just to avoid confusion.
How about
Note that this attribute is always present, even when the architecture doesn't support guests with multiple CPU dies.
The host topology info says nothing about what's possible in the guest. The only requirement for the guest is that QEMU is new enough to configure this. Aside from that any combo of sockets/ clusters/dies/cores/threads can be configured for the guest, regardless of what host topology reports.
That's not correct. Even the latest version of QEMU only allows configuring multiple dies and clusters for certain architectures.
Yes but that's a constraint of the guest machine type choice, related to the host capabilities info.
AFAICT there's no x86_64 (or ppc64, or s390x, or ...) machine type that can use CPU clusters.
I believe x86 should be gaining CPU cluster support in QEMU, though the current patch is suggesting to call them 'modules' when configuring the guest, just to absolutely maximise confusion when linux calls them 'clusters'.
So if you're planning to run a same-architecture VM (most common use case) you are not going to be able to use CPU clusters, which might be confusing considering that CPU clusters appear to be supported by your host CPU according to the capabilities XML.
On x86 AMD CPUs will report multiple dies while new Intel CPUs will report multiple clusters. In both cases, you'll be able to run a guest with multiple dies, if desire, and in future, with multiple clusters. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Feb 08, 2024 at 04:22:11PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 08:13:26AM -0800, Andrea Bolognani wrote:
AFAICT there's no x86_64 (or ppc64, or s390x, or ...) machine type that can use CPU clusters.
I believe x86 should be gaining CPU cluster support in QEMU, though the current patch is suggesting to call them 'modules' when configuring the guest, just to absolutely maximise confusion when linux calls them 'clusters'.
Lovely. Any chance we're still in time to prevent that specific trainwreck from happening?
So if you're planning to run a same-architecture VM (most common use case) you are not going to be able to use CPU clusters, which might be confusing considering that CPU clusters appear to be supported by your host CPU according to the capabilities XML.
On x86 AMD CPUs will report multiple dies while new Intel CPUs will report multiple clusters. In both cases, you'll be able to run a guest with multiple dies, if desire, and in future, with multiple clusters.
Are you aware of similar plans on the Arm side, to allow the use of multiple dies for guests? What about ppc64, s390x, riscv64, ...? Let's try one more time: Note that, even if this attribute is present, you might not be able to define guests with multiple CPU dies. This should accurately describe both today's situation, as well as whatever might come in the future to whatever architecture. -- Andrea Bolognani / Red Hat / Virtualization
On Thu, Feb 08, 2024 at 08:35:12AM -0800, Andrea Bolognani wrote:
On Thu, Feb 08, 2024 at 04:22:11PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 08:13:26AM -0800, Andrea Bolognani wrote:
AFAICT there's no x86_64 (or ppc64, or s390x, or ...) machine type that can use CPU clusters.
I believe x86 should be gaining CPU cluster support in QEMU, though the current patch is suggesting to call them 'modules' when configuring the guest, just to absolutely maximise confusion when linux calls them 'clusters'.
Lovely. Any chance we're still in time to prevent that specific trainwreck from happening?
I'm trying to discourage it, though their latest reply actually indicates they want to report 'clusters' and 'modules' as distinct concepts, so we've got even more to do on libvirt side :-) https://lists.nongnu.org/archive/html/qemu-devel/2024-02/msg00052.html
So if you're planning to run a same-architecture VM (most common use case) you are not going to be able to use CPU clusters, which might be confusing considering that CPU clusters appear to be supported by your host CPU according to the capabilities XML.
On x86 AMD CPUs will report multiple dies while new Intel CPUs will report multiple clusters. In both cases, you'll be able to run a guest with multiple dies, if desire, and in future, with multiple clusters.
Are you aware of similar plans on the Arm side, to allow the use of multiple dies for guests? What about ppc64, s390x, riscv64, ...?
Honestly don't know, its all a matter for what HW vendors choose todo.
Let's try one more time:
Note that, even if this attribute is present, you might not be able to define guests with multiple CPU dies.
This should accurately describe both today's situation, as well as whatever might come in the future to whatever architecture.
Sure. Mostly what this points out is that we don't have any reporting of available topology features in domain capabilities. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Feb 08, 2024 at 04:49:44PM +0000, Daniel P. Berrangé wrote:
On Thu, Feb 08, 2024 at 08:35:12AM -0800, Andrea Bolognani wrote:
AFAICT there's no x86_64 (or ppc64, or s390x, or ...) machine type that can use CPU clusters.
I believe x86 should be gaining CPU cluster support in QEMU, though the current patch is suggesting to call them 'modules' when configuring the guest, just to absolutely maximise confusion when linux calls them 'clusters'.
Lovely. Any chance we're still in time to prevent that specific trainwreck from happening?
I'm trying to discourage it, though their latest reply actually indicates they want to report 'clusters' and 'modules' as distinct concepts, so we've got even more to do on libvirt side :-)
https://lists.nongnu.org/archive/html/qemu-devel/2024-02/msg00052.html
It doesn't sound quite as bad then. Better than overloading an existing term with new meaning, that's for sure. The whole CPU topology hierarchy has gotten way beyond silly a long time ago anyway :) -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé