Re: [libvirt] vCPU hotplug roadmap (was: Minutes for KVM call 2013-01-15)

Am 15.01.2013 17:16, schrieb Juan Quintela: > > * cpu hot plug > - use qdev propierties conected to a set of socket objects (anthony) > - cpusets are the wrong interface (anthony) > - make a link between cpu <-> socket instead of a propierty? > - how far are we from being able to describe a cpu with -device? > (didn't heare the answer, andreas?) > - perhaps the best approach? > - After soft-freeze, exceptions depend on the maintainer > - After hard-freeze, no exceptions > -device don't require a bus, just an implementation detail, we can change that > - use cpuset as an intermediate step until full vision is implemented > - several approaches from where we are now, to have something before > we get a full solution > > > At this point, Andreas agreed to write a better summary of the > discussion and suggestions O:-) Got buried, here we go: == vCPU hot-plug user interfaces == === cpu_set === Previously available in qemu-kvm.git: `cpu_set <n+1> online` via HMP Pros: * Hides QOM/qdev implementation details (afaerber) * Thus: Doesn't depend on QOM CPUState refactoring (imammedo) * Opens a fast route to implementing vCPU unplug in KVM (imammedo) * Unintrusive to add and easy to obsolete/remove in future (imammedo) * Existing virt-test cases (afaerber) * Supported by libvirt (imammedo) * Prevents confusing guests by hot-plugging random mix of CPUs (agraf) Cons: * Cannot express topologies (ehabkost)

=== device_add === `device_add driver=Haswell-x86_64-cpu id=qdevid` [You can try this today and see it failing / not working.] Pros: * QMP/HMP command available today and known to users (afaerber) * Unified command for device and CPU hot-plug (imammedo) * Would allow first doing thread-level vCPU hotplug (imammedo) * Could be extended to support socket-level hot-plug (aliguori/imammedo) Cons: * Operates on raw QOM type name unlike -cpu (afaerber) * Needs support in libvirt for device_add driver=<CPU> (imammedo) * libvirt needs means to enumerate CPU types (imammedo) => QMP? (AF) Challenges: * No CPU qbus (afaerber) => "should work" without (aliguori) * CPU subclasses needed for identifying type name (afaerber/imammedo) => "Haswell-x86_64-cpu" does not exist yet, just "x86_64-cpu" * CPU class_init for -cpu host requires KVM init (imammedo) [suggestion by ehabkost to use kvm_arch_vcpu_init, WIP by afaerber]

* Conversion of CPU features to static properties needed (imammedo) => device_add driver=foo,level=x,xlevel=y,... * Alternatively conversion to global properties (imammedo) * Cements type names - rename for 1.4? (afaerber) => permissable (alig.) [patches for arm, m68k, openrisc, unicore32 on list] === qom-set === `qom-set` via QMP w/ link<CPUSocket> property (aliguori) Topology represented in QOM: CPUSocket has-a CPUCore has-a CPUThread a.k.a. CPUState, or CPUSocket links-to CPUCore links-to CPUThread a.k.a. CPUState Challenges (afaerber): * No CPUSocket/CPUCore objects yet and may take a while to get there... topology fields being moved to CPUState for 1.4 [done, more WIP] * No decisions on canonical paths for CPUs: CPU? machine? unassigned? * Duality of thread-level device types and socket-level? (afaerber) => fine to have, e.g., quad-core Xeon 500 device (aliguori) * CPUState is no_user (afaerber) => need to generally drop no_user for QOM (aliguori)

=== libvirt === libvirt's XML topology modelling is closer to today's -smp than to the desired QOM modelling: http://www.libvirt.org/formatcaps.html `virsh setvcpus <domain> <n>` http://libvirt.org/sources/virshcmdref/html/sect-setvcpus.html == qom-cpu course of action (afaerber) == It was requested to have vCPU hot-plug in v1.5. For device_add we need to move code from cpu_init() into QOM facilities. => QOM realize support would help [applied by aliguori] => cleanups piggy-backed onto CPU realizefn [applied to qom-cpu-next] Agreement on goal of X86CPU subclasses, but conflicts how to get there: * Refactor x86_def_t to X86CPUInfo for X86CPUClass class_init? (AF 2012) * Refactor x86_def_t to X86CPU instance_init as done for arm? * Refactor x86_def_t to class_inits? (afaerber) -> heavy merge conflicts due to bug fixes / cleanups Pro: We can get things into a consistent QOM'ish state across targets. Con: We will refactor again on top for machine-compat properties. * Keep x86_def_t within X86CPUClass as done for ppc? (WIP: afaerber) => smallest common denominator, separates x86 from cross-target work APIC ID topology fixes are being reviewed for 1.4. [merged] X86CPU wave 4 cleanups by Igor are being reviewed for 1.4. [merged] Rename CPU types according to unified <name>-<arch>-cpu scheme for 1.4? (aliguori: permissable) [patches on list] VMState series by Juan being rebased - subset for 1.4, rest for 1.5. [1.4 part on list, WIP for 1.5] Remainder is considered 1.5 material, qom-cpu-next avail. during Freeze. == Common issues (imammedo) == - back-port CPU hot-plug ACPI notification - hot-plug is not allowed on SysBus: - APIC that is created by CPU on hot-plug is a SysBus device, so essentially APIC is hot-plugged to SysBus anyway which causes abort in qdev_try_create() -> qdev_set_parent_bus(SysBus) -> bus_add_child() [no hot-plug on SysBus], which is hard to fix without allowing hot-plug on SysBus. [AF: change the APIC to not be a SysBusDevice (like CPUState) then?] - qdev_device_add() and qdev_try_create() assign SysBus by default to device if no explicit bus was provided. - qbus_find_recursive() in qdev_device_add() always returns SysBus if last 2 args are NULL, which is case with busless CPU - qdev_try_create() adds device SysBus if no bus was provided in 1st arg - vapic is not working with hot-plug in current master (do not know why yet) Experimental CPU hot-add tree using device_add implemented only for qemu64 CPU: https://github.com/imammedo/qemu/tree/x86-cpu-hot-add.WIP --- That completes Igor's and my notes, amended by status updates of mine. Sorry for the delay, I did try to address some of the discussed issues in form of patches in the meantime though. ;) CC'ing s390x, ppc and arm KVM folks that were not on the call for feedback on their requirements and thoughts. Regards, Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg