qemu: add configurable vCPU hotplug selection order Count-based vCPU hotplug currently selects eligible vCPU hotplug entities according to their vCPU IDs. For example: virsh setvcpus VM COUNT --live does not provide a way for an administrator to control which eligible entities are added or removed first. The existing ID-based policy is suitable for simple guests and for deployments where the vCPU ID layout already represents the desired placement policy. Domains that do not configure the new feature continue to use the current ID-based behavior. One use case is preserving a balanced guest NUMA CPU distribution during count-based vCPU changes. Depending on how vCPU IDs are assigned to guest NUMA cells, repeatedly adding or removing vCPUs by ID may concentrate the change in one cell before affecting another, resulting in an undesirable asymmetric guest CPU layout. There is no single balancing policy that is correct for every guest. An administrator may want to remove one hotplug entity from each guest NUMA cell in a round-robin sequence. For a guest with a threaded CPU topology, the preferred policy may instead remove a complete core, or several threads from one cell, before proceeding to the next. The appropriate sequence depends on the guest topology, workload, architecture, and performance requirements. The mechanism is not limited to NUMA balancing. It can also express administrator-defined policies such as: * retaining CPUs used for latency-sensitive, isolated, or housekeeping workloads; * implementing CPU capacity tiers; * satisfying software licensing or application-affinity constraints; * preserving a preferred socket, core, or thread population while reducing capacity; or * restoring vCPUs in a predictable order after an earlier reduction. Rather than embedding any one placement policy in libvirt, this proposal provides a general ordering mechanism. Administrators can describe the desired selection sequence in the domain XML according to the guest and deployment requirements. The existing per-vCPU 'order' attribute has different semantics and cannot be reused for this purpose. It is used to define the order and grouping of vCPUs when they are enabled. It applies to online vCPUs and is not a policy for selecting offline hotpluggable vCPUs. In particular, 'order' is not used to choose which vCPUs are removed by a count-based `setvcpus` reduction. Therefore, it cannot express an administrator-defined hot-unplug sequence. In contrast, 'hporder' is configured on disabled hotpluggable vCPUs and is used when libvirt selects eligible hotplug entities for count-based vCPU changes. It determines both which entities are removed during reduction and the order in which ranked entities are restored during expansion. It does not change the existing 'order' semantics, startup behavior, or hotplug entity composition. Introduce an optional positive per-vCPU 'hporder' XML attribute for QEMU domains. If the attribute is omitted, the vCPU remains unranked. All vCPUs belonging to the same QEMU hotplug entity must have the same 'hporder' value because they cannot be added or removed independently. The configuration is rejected if the values within an entity differ. For count-based operations, the QEMU driver applies the following selection policy: * On expansion, select unranked entities in ascending vCPU ID order, followed by ranked entities in ascending 'hporder' order. * On reduction, select ranked entities in descending 'hporder' order, followed by unranked entities in descending vCPU ID order. The ordering has two useful properties. First, domains without 'hporder' retain the existing behavior. Second, among ranked entities, expansion naturally restores entities in the reverse order in which reduction removed them. Unranked entities are selected first during expansion so that configuring 'hporder' can be used primarily to describe a preferred removal order without changing the normal expansion order of other hotpluggable vCPUs. The setting affects only the selection of eligible entities during count-based vCPU operations. It does not change: * the initial boot CPU order; * the existing per-vCPU 'order' attribute; * the composition of a QEMU hotplug entity; * whether a vCPU is hotpluggable; or * operations that explicitly address individual vCPUs. The value is preserved in domain XML and exposed through: virDomainSetVcpuHpOrder() A positive API value configures the order. Passing zero removes the configured value. Zero is not valid in domain XML and is therefore omitted when the XML is formatted. The corresponding virsh commands are: setvcpuhporder DOMAIN VCPU HPORDER delvcpuhporder DOMAIN VCPU Proposed implementation: 1. Add the 'hporder' attribute to the per-vCPU domain XML. 2. Validate that all vCPUs belonging to the same QEMU hotplug entity use the same 'hporder' value. 3. Use 'hporder' when selecting eligible vCPU hotplug entities for count-based expansion and reduction. 4. Add a public API for configuring or removing the 'hporder' value. 5. Add virsh commands for setting and removing the value. 6. Update the domain XML and API documentation to describe the new attribute, its valid values, its interaction with the existing per-vCPU 'order' attribute, and the selection rules used for expansion and reduction. 7. Add automated test coverage for XML parsing and formatting, invalid zero values, hotplug-entity consistency, and count-based expansion and reduction ordering. RFC: Feedback is welcome on the ordering policy and interface. In particular, feedback is requested on the XML/API naming, the handling of unranked vCPUs during expansion, and whether this should remain a QEMU-only feature. Signed-off-by: Partha Sarathi Satapathy <partha.satapathy@oracle.com>