[RFC] qemu: add configurable vCPU hotplug selection order
qemu: add configurable vCPU hotplug selection order Count-based vCPU hotplug currently selects eligible vCPU hotplug entities according to their vCPU IDs. For example: virsh setvcpus VM COUNT --live does not provide a way for an administrator to control which eligible entities are added or removed first. The existing ID-based policy is suitable for simple guests and for deployments where the vCPU ID layout already represents the desired placement policy. Domains that do not configure the new feature continue to use the current ID-based behavior. One use case is preserving a balanced guest NUMA CPU distribution during count-based vCPU changes. Depending on how vCPU IDs are assigned to guest NUMA cells, repeatedly adding or removing vCPUs by ID may concentrate the change in one cell before affecting another, resulting in an undesirable asymmetric guest CPU layout. There is no single balancing policy that is correct for every guest. An administrator may want to remove one hotplug entity from each guest NUMA cell in a round-robin sequence. For a guest with a threaded CPU topology, the preferred policy may instead remove a complete core, or several threads from one cell, before proceeding to the next. The appropriate sequence depends on the guest topology, workload, architecture, and performance requirements. The mechanism is not limited to NUMA balancing. It can also express administrator-defined policies such as: * retaining CPUs used for latency-sensitive, isolated, or housekeeping workloads; * implementing CPU capacity tiers; * satisfying software licensing or application-affinity constraints; * preserving a preferred socket, core, or thread population while reducing capacity; or * restoring vCPUs in a predictable order after an earlier reduction. Rather than embedding any one placement policy in libvirt, this proposal provides a general ordering mechanism. Administrators can describe the desired selection sequence in the domain XML according to the guest and deployment requirements. The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs. In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence. In contrast, 'hporder' is configured on disabled hotpluggable vCPUs and is used when libvirt selects eligible hotplug entities for count-based vCPU changes. It determines both which entities are removed during reduction and the order in which ranked entities are restored during expansion. It does not change the existing 'order' semantics, startup behavior, or hotplug entity composition. Introduce an optional positive per-vCPU 'hporder' XML attribute for QEMU domains. If the attribute is omitted, the vCPU remains unranked. All vCPUs belonging to the same QEMU hotplug entity must have the same 'hporder' value because they cannot be added or removed independently. The configuration is rejected if the values within an entity differ. For count-based operations, the QEMU driver applies the following selection policy: * On expansion, select unranked entities in ascending vCPU ID order, followed by ranked entities in ascending 'hporder' order. * On reduction, select ranked entities in descending 'hporder' order, followed by unranked entities in descending vCPU ID order. The ordering has two useful properties. First, domains without 'hporder' retain the existing behavior. Second, among ranked entities, expansion naturally restores entities in the reverse order in which reduction removed them. Unranked entities are selected first during expansion so that configuring 'hporder' can be used primarily to describe a preferred removal order without changing the normal expansion order of other hotpluggable vCPUs. The setting affects only the selection of eligible entities during count-based vCPU operations. It does not change: * the initial boot CPU order; * the existing per-vCPU 'order' attribute; * the composition of a QEMU hotplug entity; * whether a vCPU is hotpluggable; or * operations that explicitly address individual vCPUs. The value is preserved in domain XML and exposed through: virDomainSetVcpuHpOrder() A positive API value configures the order. Passing zero removes the configured value. Zero is not valid in domain XML and is therefore omitted when the XML is formatted. The corresponding virsh commands are: setvcpuhporder DOMAIN VCPU HPORDER delvcpuhporder DOMAIN VCPU Proposed implementation: 1. Add the 'hporder' attribute to the per-vCPU domain XML. 2. Validate that all vCPUs belonging to the same QEMU hotplug entity use the same 'hporder' value. 3. Use 'hporder' when selecting eligible vCPU hotplug entities for count-based expansion and reduction. 4. Add a public API for configuring or removing the 'hporder' value. 5. Add virsh commands for setting and removing the value. 6. Update the domain XML and API documentation to describe the new attribute, its valid values, its interaction with the existing per-vCPU 'order' attribute, and the selection rules used for expansion and reduction. 7. Add automated test coverage for XML parsing and formatting, invalid zero values, hotplug-entity consistency, and count-based expansion and reduction ordering. RFC: Feedback is welcome on the ordering policy and interface. In particular, feedback is requested on the XML/API naming, the handling of unranked vCPUs during expansion, and whether this should remain a QEMU-only feature. Signed-off-by: Partha Sarathi Satapathy <partha.satapathy@oracle.com>
Hi, Below are the proof-of-concept results for the proposed configurable vCPU hotplug selection order. The test uses a guest with 32 vCPUs and two guest NUMA nodes: * NUMA node 0: vCPUs 0-15 * NUMA node 1: vCPUs 16-31 The guest CPU topology is: * 2 sockets * 8 cores per socket * 2 threads per core Relevant domain XML: <vcpu placement='static'>32</vcpu> <vcpus> <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/> <vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/> <vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/> <vcpu id='3' enabled='yes' hotpluggable='yes' order='4'/> <vcpu id='4' enabled='yes' hotpluggable='yes' order='5'/> <vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/> <vcpu id='6' enabled='yes' hotpluggable='yes' order='7'/> <vcpu id='7' enabled='yes' hotpluggable='yes' order='8'/> <vcpu id='8' enabled='yes' hotpluggable='yes' order='9'/> <vcpu id='9' enabled='yes' hotpluggable='yes' order='10'/> <vcpu id='10' enabled='yes' hotpluggable='yes' order='11'/> <vcpu id='11' enabled='yes' hotpluggable='yes' order='12'/> <vcpu id='12' enabled='yes' hotpluggable='yes' order='13'/> <vcpu id='13' enabled='yes' hotpluggable='yes' order='14'/> <vcpu id='14' enabled='yes' hotpluggable='yes' order='15'/> <vcpu id='15' enabled='yes' hotpluggable='yes' order='16'/> <vcpu id='16' enabled='yes' hotpluggable='yes' order='17'/> <vcpu id='17' enabled='yes' hotpluggable='yes' order='18'/> <vcpu id='18' enabled='yes' hotpluggable='yes' order='19'/> <vcpu id='19' enabled='yes' hotpluggable='yes' order='20'/> <vcpu id='20' enabled='yes' hotpluggable='yes' order='21'/> <vcpu id='21' enabled='yes' hotpluggable='yes' order='22'/> <vcpu id='22' enabled='yes' hotpluggable='yes' order='23'/> <vcpu id='23' enabled='yes' hotpluggable='yes' order='24'/> <vcpu id='24' enabled='yes' hotpluggable='yes' order='25'/> <vcpu id='25' enabled='yes' hotpluggable='yes' order='26'/> <vcpu id='26' enabled='yes' hotpluggable='yes' order='27'/> <vcpu id='27' enabled='yes' hotpluggable='yes' order='28'/> <vcpu id='28' enabled='yes' hotpluggable='yes' order='29'/> <vcpu id='29' enabled='yes' hotpluggable='yes' order='30'/> <vcpu id='30' enabled='yes' hotpluggable='yes' order='31'/> <vcpu id='31' enabled='yes' hotpluggable='yes' order='32'/> </vcpus> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='2' dies='1' clusters='1' cores='8' threads='2'/> <cache mode='passthrough'/> <maxphysaddr mode='passthrough'/> <feature policy='require' name='arch-capabilities'/> <numa> <cell id='0' cpus='0-15' memory='20971520' unit='KiB'/> <cell id='1' cpus='16-31' memory='20971520' unit='KiB'/> </numa> </cpu> Initial state: # virsh vcpucount testvm maximum config 32 maximum live 32 current config 32 current live 32 Guest view : NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 # Existing count-based behavior Reducing the guest from 32 to 28 vCPUs: # virsh setvcpus testvm 28 --live --config causes the highest-numbered vCPUs to be removed: smpboot: CPU 31 is now offline smpboot: CPU 30 is now offline smpboot: CPU 29 is now offline smpboot: CPU 28 is now offline The resulting guest topology is: NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-27 This leaves 16 vCPUs in NUMA node 0 and 12 vCPUs in NUMA node 1. The current count-based interface therefore creates an asymmetric guest NUMA CPU distribution when vCPU IDs are grouped by NUMA node. An administrator can avoid this by using direct per-vCPU operations such as 'virsh setvcpu' and selecting the exact CPUs to remove. However, doing this requires the management layer to understand each guest's topology and calculate every individual vCPU operation. This requires additional coordination and does not scale well across many VMs and hosts. Restoring the guest to 32 vCPUs adds the same vCPUs back: # virsh setvcpus testvm 32 --live --config ACPI: CPU28 has been hot-added ACPI: CPU29 has been hot-added ACPI: CPU30 has been hot-added ACPI: CPU31 has been hot-added The guest NUMA topology returns to: NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 # Behavior with hporder For the proof of concept, the following hotplug order was configured: # virsh setvcpuhporder testvm 31 32 --live --config # virsh setvcpuhporder testvm 30 31 --live --config # virsh setvcpuhporder testvm 15 30 --live --config # virsh setvcpuhporder testvm 14 29 --live --config # virsh setvcpuhporder testvm 29 28 --live --config # virsh setvcpuhporder testvm 28 27 --live --config # virsh setvcpuhporder testvm 13 26 --live --config # virsh setvcpuhporder testvm 12 25 --live --config The resulting XML contains: # virsh dumpxml testvm | grep hporder <vcpu id='12' enabled='yes' hotpluggable='yes' order='13' hporder='25'/> <vcpu id='13' enabled='yes' hotpluggable='yes' order='14' hporder='26'/> <vcpu id='14' enabled='yes' hotpluggable='yes' order='15' hporder='29'/> <vcpu id='15' enabled='yes' hotpluggable='yes' order='16' hporder='30'/> <vcpu id='28' enabled='yes' hotpluggable='yes' order='29' hporder='27'/> <vcpu id='29' enabled='yes' hotpluggable='yes' order='30' hporder='28'/> <vcpu id='30' enabled='yes' hotpluggable='yes' order='31' hporder='31'/> <vcpu id='31' enabled='yes' hotpluggable='yes' order='32' hporder='32'/> Initial vCPU count: # virsh vcpucount testvm maximum config 32 maximum live 32 current config 32 current live 32 Guest view: NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 Reducing the guest from 32 to 28 vCPUs: # virsh setvcpus testvm 28 --live --config # virsh vcpucount testvm maximum config 32 maximum live 32 current config 28 current live 28 removes the configured vCPUs in descending hporder: smpboot: CPU 31 is now offline smpboot: CPU 30 is now offline smpboot: CPU 15 is now offline smpboot: CPU 14 is now offline The resulting guest topology is: NUMA node(s): 2 NUMA node0 CPU(s): 0-13 NUMA node1 CPU(s): 16-29 Each NUMA node now contains 14 online vCPUs. The selected CPUs are removed according to the configured priority. Reducing the guest further to 26 vCPUs: # virsh setvcpus testvm 26 --live --config # virsh vcpucount testvm maximum config 32 maximum live 32 current config 26 current live 26 removes the next configured vCPUs: smpboot: CPU 29 is now offline smpboot: CPU 28 is now offline This demonstrates that repeated count-based reductions continue to follow the administrator's configured selection order. Expanding the guest back to 32 vCPUs: # virsh setvcpus testvm 32 --live --config # virsh vcpucount testvm maximum config 32 maximum live 32 current config 32 current live 32 adds the removed vCPUs in reverse order: ACPI: CPU28 has been hot-added ACPI: CPU29 has been hot-added ACPI: CPU14 has been hot-added ACPI: CPU15 has been hot-added ACPI: CPU30 has been hot-added ACPI: CPU31 has been hot-added The guest returns to the original balanced topology: NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 # Summary The proof of concept demonstrates that: * the existing ID-based policy can produce an asymmetric guest NUMA CPU distribution; * hporder allows the administrator to define which eligible vCPUs are removed first during a count-based reduction; * the configured sequence can maintain a balanced guest NUMA CPU distribution; * repeated reductions continue to follow the configured priority; and * a later expansion restores the ranked vCPUs in the reverse order of their removal. The same mechanism can support policies other than NUMA balancing. The ordering remains administrator-defined and does not require libvirt to implement a specific topology-balancing policy. Regards, Partha
On Wed, Jul 15, 2026 at 14:56:50 +0530, Partha Satapathy via Devel wrote:
qemu: add configurable vCPU hotplug selection order
Count-based vCPU hotplug currently selects eligible vCPU hotplug entities according to their vCPU IDs. For example:
virsh setvcpus VM COUNT --live
does not provide a way for an administrator to control which eligible entities are added or removed first.
'virsh setvcpus' hotplugs the cpus in order from beginning. If you want to hotplug or hotunplug arbitrary vCPU you can use 'virsh setvcpu --enable --vcpulist $ID --domain $DOMNAME' and unplug it via '--disable'. Since we have a way to hotplug vCPUs I don't think we need to increase the complexity of the API which is meant to be simple.
The existing ID-based policy is suitable for simple guests and for deployments where the vCPU ID layout already represents the desired placement policy. Domains that do not configure the new feature continue to use the current ID-based behavior.
One use case is preserving a balanced guest NUMA CPU distribution during count-based vCPU changes. Depending on how vCPU IDs are assigned to guest NUMA cells, repeatedly adding or removing vCPUs by ID may concentrate the change in one cell before affecting another, resulting in an undesirable asymmetric guest CPU layout.
There is no single balancing policy that is correct for every guest. An administrator may want to remove one hotplug entity from each guest NUMA cell in a round-robin sequence. For a guest with a threaded CPU topology, the preferred policy may instead remove a complete core, or several threads from one cell, before proceeding to the next. The appropriate sequence depends on the guest topology, workload, architecture, and performance requirements.
The mechanism is not limited to NUMA balancing. It can also express administrator-defined policies such as:
* retaining CPUs used for latency-sensitive, isolated, or housekeeping workloads; * implementing CPU capacity tiers; * satisfying software licensing or application-affinity constraints; * preserving a preferred socket, core, or thread population while reducing capacity; or * restoring vCPUs in a predictable order after an earlier reduction.
Rather than embedding any one placement policy in libvirt, this proposal provides a general ordering mechanism. Administrators can describe the desired selection sequence in the domain XML according to the guest and deployment requirements.
The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs.
In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence.
In contrast, 'hporder' is configured on disabled hotpluggable vCPUs and is used when libvirt selects eligible hotplug entities for count-based vCPU changes. It determines both which entities are removed during reduction and the order in which ranked entities are restored during expansion. It does not change the existing 'order' semantics, startup behavior, or hotplug entity composition.
Introduce an optional positive per-vCPU 'hporder' XML attribute for QEMU domains. If the attribute is omitted, the vCPU remains unranked.
All vCPUs belonging to the same QEMU hotplug entity must have the same 'hporder' value because they cannot be added or removed independently. The configuration is rejected if the values within an entity differ.
For count-based operations, the QEMU driver applies the following selection policy:
* On expansion, select unranked entities in ascending vCPU ID order, followed by ranked entities in ascending 'hporder' order.
* On reduction, select ranked entities in descending 'hporder' order, followed by unranked entities in descending vCPU ID order.
The ordering has two useful properties. First, domains without 'hporder' retain the existing behavior. Second, among ranked entities, expansion naturally restores entities in the reverse order in which reduction removed them.
Unranked entities are selected first during expansion so that configuring 'hporder' can be used primarily to describe a preferred removal order without changing the normal expansion order of other hotpluggable vCPUs.
The setting affects only the selection of eligible entities during count-based vCPU operations. It does not change:
* the initial boot CPU order; * the existing per-vCPU 'order' attribute; * the composition of a QEMU hotplug entity; * whether a vCPU is hotpluggable; or * operations that explicitly address individual vCPUs.
The value is preserved in domain XML and exposed through:
virDomainSetVcpuHpOrder()
A positive API value configures the order. Passing zero removes the configured value. Zero is not valid in domain XML and is therefore omitted when the XML is formatted.
The corresponding virsh commands are:
setvcpuhporder DOMAIN VCPU HPORDER delvcpuhporder DOMAIN VCPU
Proposed implementation:
1. Add the 'hporder' attribute to the per-vCPU domain XML. 2. Validate that all vCPUs belonging to the same QEMU hotplug entity use the same 'hporder' value. 3. Use 'hporder' when selecting eligible vCPU hotplug entities for count-based expansion and reduction. 4. Add a public API for configuring or removing the 'hporder' value. 5. Add virsh commands for setting and removing the value. 6. Update the domain XML and API documentation to describe the new attribute, its valid values, its interaction with the existing per-vCPU 'order' attribute, and the selection rules used for expansion and reduction. 7. Add automated test coverage for XML parsing and formatting, invalid zero values, hotplug-entity consistency, and count-based expansion and reduction ordering.
RFC: Feedback is welcome on the ordering policy and interface. In particular, feedback is requested on the XML/API naming, the handling of unranked vCPUs during expansion, and whether this should remain a QEMU-only feature.
Signed-off-by: Partha Sarathi Satapathy <partha.satapathy@oracle.com>
These arguments don't really seem to be enticing to add the extra complexity since we allow arbitrary hotplug. The writeup also doesn't show what happens if an user uses the specific hotplug API in combination with this.
Peter Krempa wrote:
On Wed, Jul 15, 2026 at 14:56:50 +0530, Partha Satapathy via Devel wrote:
qemu: add configurable vCPU hotplug selection order Count-based vCPU hotplug currently selects eligible vCPU hotplug entities according to their vCPU IDs. For example: virsh setvcpus VM COUNT --live does not provide a way for an administrator to control which eligible entities are added or removed first. 'virsh setvcpus' hotplugs the cpus in order from beginning. If you want to hotplug or hotunplug arbitrary vCPU you can use
'virsh setvcpu --enable --vcpulist $ID --domain $DOMNAME'
and unplug it via '--disable'.
Since we have a way to hotplug vCPUs I don't think we need to increase the complexity of the API which is meant to be simple.
Thanks for the comment. We are aware of virsh setvcpu --enable/--disable --vcpulist, and this proposal does not intend to replace it. It remains appropriate when management explicitly selects individual vCPUs. The intended use case is a management layer that scales domains by a target vCPU count while preserving an administrator-defined placement policy. Implementing this solely with setvcpu requires the management layer to inspect the domain topology and current state, select eligible vCPUs, account for QEMU hotplug-entity grouping, and issue the individual operations in the required order for every scale event. hporder allows that policy to be configured once in the domain XML. Subsequent count-based setvcpus operations can then use the existing interface while libvirt selects eligible hotplug entities according to that policy. In particular, this avoids requiring every management client to duplicate the selection logic. Domains that do not configure hporder retain the existing ID-based selection behavior. Thus the default setvcpus behavior and the per-vCPU setvcpu interface remain unchanged.
The existing ID-based policy is suitable for simple guests and for deployments where the vCPU ID layout already represents the desired placement policy. Domains that do not configure the new feature continue to use the current ID-based behavior. One use case is preserving a balanced guest NUMA CPU distribution during count-based vCPU changes. Depending on how vCPU IDs are assigned to guest NUMA cells, repeatedly adding or removing vCPUs by ID may concentrate the change in one cell before affecting another, resulting in an undesirable asymmetric guest CPU layout. There is no single balancing policy that is correct for every guest. An administrator may want to remove one hotplug entity from each guest NUMA cell in a round-robin sequence. For a guest with a threaded CPU topology, the preferred policy may instead remove a complete core, or several threads from one cell, before proceeding to the next. The appropriate sequence depends on the guest topology, workload, architecture, and performance requirements. The mechanism is not limited to NUMA balancing. It can also express administrator-defined policies such as: * retaining CPUs used for latency-sensitive, isolated, or housekeeping workloads; * implementing CPU capacity tiers; * satisfying software licensing or application-affinity constraints; * preserving a preferred socket, core, or thread population while reducing capacity; or * restoring vCPUs in a predictable order after an earlier reduction. Rather than embedding any one placement policy in libvirt, this proposal provides a general ordering mechanism. Administrators can describe the desired selection sequence in the domain XML according to the guest and deployment requirements. The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs. In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence. In contrast, 'hporder' is configured on disabled hotpluggable vCPUs and is used when libvirt selects eligible hotplug entities for count-based vCPU changes. It determines both which entities are removed during reduction and the order in which ranked entities are restored during expansion. It does not change the existing 'order' semantics, startup behavior, or hotplug entity composition. Introduce an optional positive per-vCPU 'hporder' XML attribute for QEMU domains. If the attribute is omitted, the vCPU remains unranked. All vCPUs belonging to the same QEMU hotplug entity must have the same 'hporder' value because they cannot be added or removed independently. The configuration is rejected if the values within an entity differ. For count-based operations, the QEMU driver applies the following selection policy: * On expansion, select unranked entities in ascending vCPU ID order, followed by ranked entities in ascending 'hporder' order. * On reduction, select ranked entities in descending 'hporder' order, followed by unranked entities in descending vCPU ID order. The ordering has two useful properties. First, domains without 'hporder' retain the existing behavior. Second, among ranked entities, expansion naturally restores entities in the reverse order in which reduction removed them. Unranked entities are selected first during expansion so that configuring 'hporder' can be used primarily to describe a preferred removal order without changing the normal expansion order of other hotpluggable vCPUs. The setting affects only the selection of eligible entities during count-based vCPU operations. It does not change: * the initial boot CPU order; * the existing per-vCPU 'order' attribute; * the composition of a QEMU hotplug entity; * whether a vCPU is hotpluggable; or * operations that explicitly address individual vCPUs. The value is preserved in domain XML and exposed through: virDomainSetVcpuHpOrder() A positive API value configures the order. Passing zero removes the configured value. Zero is not valid in domain XML and is therefore omitted when the XML is formatted. The corresponding virsh commands are: setvcpuhporder DOMAIN VCPU HPORDER delvcpuhporder DOMAIN VCPU Proposed implementation: 1. Add the 'hporder' attribute to the per-vCPU domain XML. 2. Validate that all vCPUs belonging to the same QEMU hotplug entity use the same 'hporder' value. 3. Use 'hporder' when selecting eligible vCPU hotplug entities for count-based expansion and reduction. 4. Add a public API for configuring or removing the 'hporder' value. 5. Add virsh commands for setting and removing the value. 6. Update the domain XML and API documentation to describe the new attribute, its valid values, its interaction with the existing per-vCPU 'order' attribute, and the selection rules used for expansion and reduction. 7. Add automated test coverage for XML parsing and formatting, invalid zero values, hotplug-entity consistency, and count-based expansion and reduction ordering. RFC: Feedback is welcome on the ordering policy and interface. In particular, feedback is requested on the XML/API naming, the handling of unranked vCPUs during expansion, and whether this should remain a QEMU-only feature. Signed-off-by: Partha Sarathi Satapathy <partha.satapathy@oracle.com> These arguments don't really seem to be enticing to add the extra complexity since we allow arbitrary hotplug.
The writeup also doesn't show what happens if an user uses the specific hotplug API in combination with this.
hporder is used only for count-based setvcpus operations, where libvirt must select eligible hotplug entities to reach the requested count. An explicit per-vCPU operation using setvcpu --enable/--disable --vcpulist selects the requested vCPU (and its corresponding hotplug entity) directly. It does not consult hporder, and hporder does not prevent or override that operation. The main goal of hporder is to allow an administrator to configure the vCPU hotplug selection policy once in the domain XML, so subsequent count-based scaling requires no further per-vCPU intervention from the administrator or management layer.
On Wed, Jul 15, 2026 at 02:56:50PM +0530, Partha Satapathy via Devel wrote:
The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs.
In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence.
Aren't you simply describing a bug in the implementation of "order". I defined a guest with # virsh dumpxml --inactive 2 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes" order="3"/> <vcpu id="3" enabled="no" hotpluggable="yes" order="4"/> <vcpu id="4" enabled="no" hotpluggable="yes" order="5"/> <vcpu id="5" enabled="no" hotpluggable="yes" order="6"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="7"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="8"/> Yet when I start this guest, libvirt throws away the order information I configured # virsh setvcpus rhel92 6 # virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes"/> <vcpu id="3" enabled="no" hotpluggable="yes"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/> and hotplugging CPUs ignores the order I defined # virsh setvcpus rhel92 6 # virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="6">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="yes" hotpluggable="yes" order="5"/> <vcpu id="3" enabled="yes" hotpluggable="yes" order="6"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/> IMHO, that is just broken and we should fix "order" to be honoured, which would appear to address your use case. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On Wed, Jul 15, 2026 at 16:11:06 +0100, Daniel P. Berrangé via Devel wrote:
On Wed, Jul 15, 2026 at 02:56:50PM +0530, Partha Satapathy via Devel wrote:
The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs.
In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence.
Aren't you simply describing a bug in the implementation of "order".
I defined a guest with
# virsh dumpxml --inactive 2 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes" order="3"/> <vcpu id="3" enabled="no" hotpluggable="yes" order="4"/> <vcpu id="4" enabled="no" hotpluggable="yes" order="5"/> <vcpu id="5" enabled="no" hotpluggable="yes" order="6"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="7"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="8"/>
Yet when I start this guest, libvirt throws away the order information I configured
# virsh setvcpus rhel92 6
# virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes"/> <vcpu id="3" enabled="no" hotpluggable="yes"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/>
and hotplugging CPUs ignores the order I defined
# virsh setvcpus rhel92 6
# virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="6">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="yes" hotpluggable="yes" order="5"/> <vcpu id="3" enabled="yes" hotpluggable="yes" order="6"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/>
IMHO, that is just broken and we should fix "order" to be honoured, which would appear to address your use case.
Order is currently the effective order that was used to hotplug the devices in so that e.g. on migration the state can be properly restored, so it's used only for active vcpus and ignored for inactive. Since we have API to hotplug specific vCPU too, this *must* update the order (in order for the migration to stay compatible) so it'd effectively change the "config" if users were able to set the order (e.g. un-set order from another vCPU if it were plugged in out of order). Effective re-use of 'order' is thus not reallistically feasible in this case. As I've said in the other thread we do allow hotplug vcpus in arbitrary order, so this would be just syntax-sugar to support hotplug in arbitrary order via the sequenital order API. I don't feel it's worth keeping this complex logic in libvirt as the use cases are relatively niche and it's a lot of extra complexity to maintain.
Peter Krempa wrote:
On Wed, Jul 15, 2026 at 16:11:06 +0100, Daniel P. Berrangé via Devel wrote:
On Wed, Jul 15, 2026 at 02:56:50PM +0530, Partha Satapathy via Devel wrote: The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs. In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence. Aren't you simply describing a bug in the implementation of "order". I defined a guest with # virsh dumpxml --inactive 2 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes" order="3"/> <vcpu id="3" enabled="no" hotpluggable="yes" order="4"/> <vcpu id="4" enabled="no" hotpluggable="yes" order="5"/> <vcpu id="5" enabled="no" hotpluggable="yes" order="6"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="7"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="8"/> Yet when I start this guest, libvirt throws away the order information I configured # virsh setvcpus rhel92 6 # virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes"/> <vcpu id="3" enabled="no" hotpluggable="yes"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/> and hotplugging CPUs ignores the order I defined # virsh setvcpus rhel92 6 # virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="6">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="yes" hotpluggable="yes" order="5"/> <vcpu id="3" enabled="yes" hotpluggable="yes" order="6"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/> IMHO, that is just broken and we should fix "order" to be honoured, which would appear to address your use case. Order is currently the effective order that was used to hotplug the devices in so that e.g. on migration the state can be properly restored, so it's used only for active vcpus and ignored for inactive.
Since we have API to hotplug specific vCPU too, this *must* update the order (in order for the migration to stay compatible) so it'd effectively change the "config" if users were able to set the order (e.g. un-set order from another vCPU if it were plugged in out of order).
Effective re-use of 'order' is thus not reallistically feasible in this case.
As I've said in the other thread we do allow hotplug vcpus in arbitrary order, so this would be just syntax-sugar to support hotplug in arbitrary order via the sequenital order API.
I don't feel it's worth keeping this complex logic in libvirt as the use cases are relatively niche and it's a lot of extra complexity to maintain.
Hi Peter, I missed your reply before my last post. I understand that arbitrary vCPU ordering can be achieved by explicitly adding or removing individual vCPUs with setvcpu. This RFC aims to make that process a lot easier. Large data-centre deployments require management layers to repeatedly calculate eligible vCPUs, hotplug groups, NUMA balance, topology, and scale direction. This becomes complex and error-prone. This RFC is an opt-in mechanism to store that selection policy once in XML. It does not impose a balancing policy or change existing behaviour. Would you be open to reconsidering whether this offers sufficient value for standardized, repeatable count-based scaling in libvirt mainline? If not, I understand. Thank you for the feedback; I will not pursue this RFC further. Regards, Partha
On Thu, Jul 16, 2026 at 10:16:07AM +0200, Peter Krempa wrote:
On Wed, Jul 15, 2026 at 16:11:06 +0100, Daniel P. Berrangé via Devel wrote:
On Wed, Jul 15, 2026 at 02:56:50PM +0530, Partha Satapathy via Devel wrote:
The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs.
In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence.
Aren't you simply describing a bug in the implementation of "order".
I defined a guest with
# virsh dumpxml --inactive 2 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes" order="3"/> <vcpu id="3" enabled="no" hotpluggable="yes" order="4"/> <vcpu id="4" enabled="no" hotpluggable="yes" order="5"/> <vcpu id="5" enabled="no" hotpluggable="yes" order="6"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="7"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="8"/>
Yet when I start this guest, libvirt throws away the order information I configured
# virsh setvcpus rhel92 6
# virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="4">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="no" hotpluggable="yes"/> <vcpu id="3" enabled="no" hotpluggable="yes"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/>
and hotplugging CPUs ignores the order I defined
# virsh setvcpus rhel92 6
# virsh dumpxml rhel92 --xpath //vcpu <vcpu placement="static" current="6">8</vcpu> <vcpu id="0" enabled="yes" hotpluggable="no" order="1"/> <vcpu id="1" enabled="yes" hotpluggable="yes" order="2"/> <vcpu id="2" enabled="yes" hotpluggable="yes" order="5"/> <vcpu id="3" enabled="yes" hotpluggable="yes" order="6"/> <vcpu id="4" enabled="no" hotpluggable="yes"/> <vcpu id="5" enabled="no" hotpluggable="yes"/> <vcpu id="6" enabled="yes" hotpluggable="yes" order="3"/> <vcpu id="7" enabled="yes" hotpluggable="yes" order="4"/>
IMHO, that is just broken and we should fix "order" to be honoured, which would appear to address your use case.
Order is currently the effective order that was used to hotplug the devices in so that e.g. on migration the state can be properly restored, so it's used only for active vcpus and ignored for inactive.
Hmmm, I'm surprised migration state cares about the order in which we hotplug the CPUs. I would have thought that if we setup a VM with 8 CPUs, and only had 6 online, the VM state ought to always reflect the max 8 and track the online state, such that libvirt just needed to make sure the same subset of CPUs are online.
Since we have API to hotplug specific vCPU too, this *must* update the order (in order for the migration to stay compatible) so it'd effectively change the "config" if users were able to set the order (e.g. un-set order from another vCPU if it were plugged in out of order).
Effective re-use of 'order' is thus not reallistically feasible in this case.
It is unfortunate that we accept the 'order' attrbiute in the inactive XML at all, if we're ignoring it and inserting new values for live VM state. That didn't seem to reflect what our domain XML docs suggest the 'order' attribute used for, which reads somewhat like it matches Partha's required use case. Our docs could be improved in this regard I think.
As I've said in the other thread we do allow hotplug vcpus in arbitrary order, so this would be just syntax-sugar to support hotplug in arbitrary order via the sequenital order API.
I don't feel it's worth keeping this complex logic in libvirt as the use cases are relatively niche and it's a lot of extra complexity to maintain.
Yeah, it is a bit of an awkward case. It also gets slightly on the wrong side of "policy" vs "mechanism", where we try to avoid defining policy and leave it to mgmt app to decide upon. I am slightly concerned that an "order" attribute may not capture policies beyond the one expressed in the use case. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
Thanks for the review, Daniel. order and the proposed hporder serve related but distinct purposes. The existing order attribute specifies the sequence in which online vCPU devices are added. During startup, libvirt uses it when realizing initially enabled hotpluggable CPU entities. The domain XML documentation also states: Hypervisors may clear or update ordering information during certain operations to assure valid configuration. The documentation permits `order` to be rewritten as the vCPU configuration changes, where required to assure a valid configuration. At startup, the QEMU driver updates the live ordering to reflect the CPU enable sequence reported by QEMU. - order describes the ordering of enabled CPU devices. Its configured use is during startup, when libvirt realizes the initially enabled hotpluggable CPU entities. - It is not a policy for selecting vCPUs to delete during a count-based decrease. - It does not provide a selection policy for offline vCPUs during a count-based increase. Offline vCPUs do not have a realized insertion order, and their order values may be cleared. - At present, there is no public interface to set a chosen order directly on a live running domain. It can be configured in persistent XML, for example with virsh edit, while live/status XML reflects the realized sequence. The last point could be addressed by adding an interface to change order in a live domain, but that alone would not define the desired count-based selection policy for offline vCPUs. The proposed per-vCPU hporder attribute is intended to provide this separate, administrator-defined policy for dynamic count-based scaling: - during a decrease, eligible hotpluggable CPU entities with the highest removal priority would be selected first; and - during a later increase, eligible disabled entities would be selected in reverse priority order. Unlike order, hporder would remain meaningful for disabled hotpluggable vCPU entities. I agree that the XML documentation could make the distinction clearer. The permission to clear or update ordering information does not explain the difference between realized online-device ordering and a persistent future scaling policy. One possible alternative would be to overload order through a domain-level policy, for example: <vcpu placement='static' current='4' orderpolicy='strict'>8</vcpu> This alternative may be less desirable, since it would allow order to represent runtime ordering in one mode and persistent future policy in another. A separate per-vCPU hporder attribute would preserve the existing order semantics while making the scaling policy explicit. For setvcpu, the caller explicitly selects the vCPU(s) to hotplug or hot-unplug, so hporder would not affect that selection. If the user selects multiple CPU entities with setvcpu, hporder could determine their processing order based on the configured values. Peter and Daniel, I would welcome feedback on the desired setvcpu behavior, as well as the XML representation and proposed API scope.
participants (4)
-
Daniel P. Berrangé -
Partha Satapathy -
partha.satapathy＠oracle.com
-
Peter Krempa