Peter Krempa wrote:
On Wed, Jul 15, 2026 at 14:56:50 +0530, Partha Satapathy via Devel wrote:
qemu: add configurable vCPU hotplug selection order Count-based vCPU hotplug currently selects eligible vCPU hotplug entities according to their vCPU IDs. For example: virsh setvcpus VM COUNT --live does not provide a way for an administrator to control which eligible entities are added or removed first. 'virsh setvcpus' hotplugs the cpus in order from beginning. If you want to hotplug or hotunplug arbitrary vCPU you can use
'virsh setvcpu --enable --vcpulist $ID --domain $DOMNAME'
and unplug it via '--disable'.
Since we have a way to hotplug vCPUs I don't think we need to increase the complexity of the API which is meant to be simple.
Thanks for the comment. We are aware of virsh setvcpu --enable/--disable --vcpulist, and this proposal does not intend to replace it. It remains appropriate when management explicitly selects individual vCPUs. The intended use case is a management layer that scales domains by a target vCPU count while preserving an administrator-defined placement policy. Implementing this solely with setvcpu requires the management layer to inspect the domain topology and current state, select eligible vCPUs, account for QEMU hotplug-entity grouping, and issue the individual operations in the required order for every scale event. hporder allows that policy to be configured once in the domain XML. Subsequent count-based setvcpus operations can then use the existing interface while libvirt selects eligible hotplug entities according to that policy. In particular, this avoids requiring every management client to duplicate the selection logic. Domains that do not configure hporder retain the existing ID-based selection behavior. Thus the default setvcpus behavior and the per-vCPU setvcpu interface remain unchanged.
The existing ID-based policy is suitable for simple guests and for deployments where the vCPU ID layout already represents the desired placement policy. Domains that do not configure the new feature continue to use the current ID-based behavior. One use case is preserving a balanced guest NUMA CPU distribution during count-based vCPU changes. Depending on how vCPU IDs are assigned to guest NUMA cells, repeatedly adding or removing vCPUs by ID may concentrate the change in one cell before affecting another, resulting in an undesirable asymmetric guest CPU layout. There is no single balancing policy that is correct for every guest. An administrator may want to remove one hotplug entity from each guest NUMA cell in a round-robin sequence. For a guest with a threaded CPU topology, the preferred policy may instead remove a complete core, or several threads from one cell, before proceeding to the next. The appropriate sequence depends on the guest topology, workload, architecture, and performance requirements. The mechanism is not limited to NUMA balancing. It can also express administrator-defined policies such as: * retaining CPUs used for latency-sensitive, isolated, or housekeeping workloads; * implementing CPU capacity tiers; * satisfying software licensing or application-affinity constraints; * preserving a preferred socket, core, or thread population while reducing capacity; or * restoring vCPUs in a predictable order after an earlier reduction. Rather than embedding any one placement policy in libvirt, this proposal provides a general ordering mechanism. Administrators can describe the desired selection sequence in the domain XML according to the guest and deployment requirements. The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs. In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence. In contrast, 'hporder' is configured on disabled hotpluggable vCPUs and is used when libvirt selects eligible hotplug entities for count-based vCPU changes. It determines both which entities are removed during reduction and the order in which ranked entities are restored during expansion. It does not change the existing 'order' semantics, startup behavior, or hotplug entity composition. Introduce an optional positive per-vCPU 'hporder' XML attribute for QEMU domains. If the attribute is omitted, the vCPU remains unranked. All vCPUs belonging to the same QEMU hotplug entity must have the same 'hporder' value because they cannot be added or removed independently. The configuration is rejected if the values within an entity differ. For count-based operations, the QEMU driver applies the following selection policy: * On expansion, select unranked entities in ascending vCPU ID order, followed by ranked entities in ascending 'hporder' order. * On reduction, select ranked entities in descending 'hporder' order, followed by unranked entities in descending vCPU ID order. The ordering has two useful properties. First, domains without 'hporder' retain the existing behavior. Second, among ranked entities, expansion naturally restores entities in the reverse order in which reduction removed them. Unranked entities are selected first during expansion so that configuring 'hporder' can be used primarily to describe a preferred removal order without changing the normal expansion order of other hotpluggable vCPUs. The setting affects only the selection of eligible entities during count-based vCPU operations. It does not change: * the initial boot CPU order; * the existing per-vCPU 'order' attribute; * the composition of a QEMU hotplug entity; * whether a vCPU is hotpluggable; or * operations that explicitly address individual vCPUs. The value is preserved in domain XML and exposed through: virDomainSetVcpuHpOrder() A positive API value configures the order. Passing zero removes the configured value. Zero is not valid in domain XML and is therefore omitted when the XML is formatted. The corresponding virsh commands are: setvcpuhporder DOMAIN VCPU HPORDER delvcpuhporder DOMAIN VCPU Proposed implementation: 1. Add the 'hporder' attribute to the per-vCPU domain XML. 2. Validate that all vCPUs belonging to the same QEMU hotplug entity use the same 'hporder' value. 3. Use 'hporder' when selecting eligible vCPU hotplug entities for count-based expansion and reduction. 4. Add a public API for configuring or removing the 'hporder' value. 5. Add virsh commands for setting and removing the value. 6. Update the domain XML and API documentation to describe the new attribute, its valid values, its interaction with the existing per-vCPU 'order' attribute, and the selection rules used for expansion and reduction. 7. Add automated test coverage for XML parsing and formatting, invalid zero values, hotplug-entity consistency, and count-based expansion and reduction ordering. RFC: Feedback is welcome on the ordering policy and interface. In particular, feedback is requested on the XML/API naming, the handling of unranked vCPUs during expansion, and whether this should remain a QEMU-only feature. Signed-off-by: Partha Sarathi Satapathy <partha.satapathy@oracle.com> These arguments don't really seem to be enticing to add the extra complexity since we allow arbitrary hotplug.
The writeup also doesn't show what happens if an user uses the specific hotplug API in combination with this.
hporder is used only for count-based setvcpus operations, where libvirt must select eligible hotplug entities to reach the requested count. An explicit per-vCPU operation using setvcpu --enable/--disable --vcpulist selects the requested vCPU (and its corresponding hotplug entity) directly. It does not consult hporder, and hporder does not prevent or override that operation. The main goal of hporder is to allow an administrator to configure the vCPU hotplug selection policy once in the domain XML, so subsequent count-based scaling requires no further per-vCPU intervention from the administrator or management layer.