Hi, Below are the proof-of-concept results for the proposed configurable vCPU hotplug selection order. The test uses a guest with 32 vCPUs and two guest NUMA nodes: * NUMA node 0: vCPUs 0-15 * NUMA node 1: vCPUs 16-31 The guest CPU topology is: * 2 sockets * 8 cores per socket * 2 threads per core Relevant domain XML: <vcpu placement='static'>32</vcpu> <vcpus> <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/> <vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/> <vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/> <vcpu id='3' enabled='yes' hotpluggable='yes' order='4'/> <vcpu id='4' enabled='yes' hotpluggable='yes' order='5'/> <vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/> <vcpu id='6' enabled='yes' hotpluggable='yes' order='7'/> <vcpu id='7' enabled='yes' hotpluggable='yes' order='8'/> <vcpu id='8' enabled='yes' hotpluggable='yes' order='9'/> <vcpu id='9' enabled='yes' hotpluggable='yes' order='10'/> <vcpu id='10' enabled='yes' hotpluggable='yes' order='11'/> <vcpu id='11' enabled='yes' hotpluggable='yes' order='12'/> <vcpu id='12' enabled='yes' hotpluggable='yes' order='13'/> <vcpu id='13' enabled='yes' hotpluggable='yes' order='14'/> <vcpu id='14' enabled='yes' hotpluggable='yes' order='15'/> <vcpu id='15' enabled='yes' hotpluggable='yes' order='16'/> <vcpu id='16' enabled='yes' hotpluggable='yes' order='17'/> <vcpu id='17' enabled='yes' hotpluggable='yes' order='18'/> <vcpu id='18' enabled='yes' hotpluggable='yes' order='19'/> <vcpu id='19' enabled='yes' hotpluggable='yes' order='20'/> <vcpu id='20' enabled='yes' hotpluggable='yes' order='21'/> <vcpu id='21' enabled='yes' hotpluggable='yes' order='22'/> <vcpu id='22' enabled='yes' hotpluggable='yes' order='23'/> <vcpu id='23' enabled='yes' hotpluggable='yes' order='24'/> <vcpu id='24' enabled='yes' hotpluggable='yes' order='25'/> <vcpu id='25' enabled='yes' hotpluggable='yes' order='26'/> <vcpu id='26' enabled='yes' hotpluggable='yes' order='27'/> <vcpu id='27' enabled='yes' hotpluggable='yes' order='28'/> <vcpu id='28' enabled='yes' hotpluggable='yes' order='29'/> <vcpu id='29' enabled='yes' hotpluggable='yes' order='30'/> <vcpu id='30' enabled='yes' hotpluggable='yes' order='31'/> <vcpu id='31' enabled='yes' hotpluggable='yes' order='32'/> </vcpus> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='2' dies='1' clusters='1' cores='8' threads='2'/> <cache mode='passthrough'/> <maxphysaddr mode='passthrough'/> <feature policy='require' name='arch-capabilities'/> <numa> <cell id='0' cpus='0-15' memory='20971520' unit='KiB'/> <cell id='1' cpus='16-31' memory='20971520' unit='KiB'/> </numa> </cpu> Initial state: # virsh vcpucount testvm maximum config 32 maximum live 32 current config 32 current live 32 Guest view : NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 # Existing count-based behavior Reducing the guest from 32 to 28 vCPUs: # virsh setvcpus testvm 28 --live --config causes the highest-numbered vCPUs to be removed: smpboot: CPU 31 is now offline smpboot: CPU 30 is now offline smpboot: CPU 29 is now offline smpboot: CPU 28 is now offline The resulting guest topology is: NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-27 This leaves 16 vCPUs in NUMA node 0 and 12 vCPUs in NUMA node 1. The current count-based interface therefore creates an asymmetric guest NUMA CPU distribution when vCPU IDs are grouped by NUMA node. An administrator can avoid this by using direct per-vCPU operations such as 'virsh setvcpu' and selecting the exact CPUs to remove. However, doing this requires the management layer to understand each guest's topology and calculate every individual vCPU operation. This requires additional coordination and does not scale well across many VMs and hosts. Restoring the guest to 32 vCPUs adds the same vCPUs back: # virsh setvcpus testvm 32 --live --config ACPI: CPU28 has been hot-added ACPI: CPU29 has been hot-added ACPI: CPU30 has been hot-added ACPI: CPU31 has been hot-added The guest NUMA topology returns to: NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 # Behavior with hporder For the proof of concept, the following hotplug order was configured: # virsh setvcpuhporder testvm 31 32 --live --config # virsh setvcpuhporder testvm 30 31 --live --config # virsh setvcpuhporder testvm 15 30 --live --config # virsh setvcpuhporder testvm 14 29 --live --config # virsh setvcpuhporder testvm 29 28 --live --config # virsh setvcpuhporder testvm 28 27 --live --config # virsh setvcpuhporder testvm 13 26 --live --config # virsh setvcpuhporder testvm 12 25 --live --config The resulting XML contains: # virsh dumpxml testvm | grep hporder <vcpu id='12' enabled='yes' hotpluggable='yes' order='13' hporder='25'/> <vcpu id='13' enabled='yes' hotpluggable='yes' order='14' hporder='26'/> <vcpu id='14' enabled='yes' hotpluggable='yes' order='15' hporder='29'/> <vcpu id='15' enabled='yes' hotpluggable='yes' order='16' hporder='30'/> <vcpu id='28' enabled='yes' hotpluggable='yes' order='29' hporder='27'/> <vcpu id='29' enabled='yes' hotpluggable='yes' order='30' hporder='28'/> <vcpu id='30' enabled='yes' hotpluggable='yes' order='31' hporder='31'/> <vcpu id='31' enabled='yes' hotpluggable='yes' order='32' hporder='32'/> Initial vCPU count: # virsh vcpucount testvm maximum config 32 maximum live 32 current config 32 current live 32 Guest view: NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 Reducing the guest from 32 to 28 vCPUs: # virsh setvcpus testvm 28 --live --config # virsh vcpucount testvm maximum config 32 maximum live 32 current config 28 current live 28 removes the configured vCPUs in descending hporder: smpboot: CPU 31 is now offline smpboot: CPU 30 is now offline smpboot: CPU 15 is now offline smpboot: CPU 14 is now offline The resulting guest topology is: NUMA node(s): 2 NUMA node0 CPU(s): 0-13 NUMA node1 CPU(s): 16-29 Each NUMA node now contains 14 online vCPUs. The selected CPUs are removed according to the configured priority. Reducing the guest further to 26 vCPUs: # virsh setvcpus testvm 26 --live --config # virsh vcpucount testvm maximum config 32 maximum live 32 current config 26 current live 26 removes the next configured vCPUs: smpboot: CPU 29 is now offline smpboot: CPU 28 is now offline This demonstrates that repeated count-based reductions continue to follow the administrator's configured selection order. Expanding the guest back to 32 vCPUs: # virsh setvcpus testvm 32 --live --config # virsh vcpucount testvm maximum config 32 maximum live 32 current config 32 current live 32 adds the removed vCPUs in reverse order: ACPI: CPU28 has been hot-added ACPI: CPU29 has been hot-added ACPI: CPU14 has been hot-added ACPI: CPU15 has been hot-added ACPI: CPU30 has been hot-added ACPI: CPU31 has been hot-added The guest returns to the original balanced topology: NUMA node(s): 2 NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 # Summary The proof of concept demonstrates that: * the existing ID-based policy can produce an asymmetric guest NUMA CPU distribution; * hporder allows the administrator to define which eligible vCPUs are removed first during a count-based reduction; * the configured sequence can maintain a balanced guest NUMA CPU distribution; * repeated reductions continue to follow the configured priority; and * a later expansion restores the ranked vCPUs in the reverse order of their removal. The same mechanism can support policies other than NUMA balancing. The ordering remains administrator-defined and does not require libvirt to implement a specific topology-balancing policy. Regards, Partha