[libvirt] libvirt<->QEMU interfaces for CPU models
Hi, After a long time trying to figure out the proper modelling inside QEMU, I believe the plans are now clearer in QEMU, so it's time to coordinate more closely with libvirt to try to make use of the new stuff. I tried to enumerate the libvirt requirements and current problems, and how we should be able to solve those problems using the X86CPU subclasses and properties, on the following wiki page: http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvi... I am pasting the section contents below, to facilitate discussion: = Ensuring predictable set of guest features = Requirement: libvirt needs to ensure all features required on the command-line are present and exposed to the guest. Current problem: libvirt doesn't use the "enforce" flag so it can't guarantee that a given feature will be actually exposed to the guest. Solution: use the "enforce" flag on the "-cpu" option. Limitation: no proper machine-friendly interface to report which features are missing. Workaround: See "querying for host capabilities" below. == Future plans == Machine-friendly reporting of missing host features/capabilities. = Listing CPU models = Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option. Current problem: libvirt relies on help output parsing for that. Solution: use QMP qom-list-types command. Dependency: X86CPU subclasses. Limitation: needs a live QEMU process for the query. Solution: use QMP query-cpu-definitions command. Limitation: needs a live QEMU process for the query. == Future plans == It would be interesting to get rid of the requirement for a live QEMU process (with a complete machine being created) to be already running. = Getting information about CPU models = Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user. Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true. Challenge: the resulting CPU features depend on lots of factors, including the machine-type. Workaround: start a paused VM and query for the CPU device information after the CPU was created. Solution: start a paused VM with no devices, but with the right machine-type and right CPU model. Use QMP QOM commands to query for CPU flags (especially the properties starting with the "f-" prefix). Dependency: X86CPU feature properties ("f-*" properties). Limitation: requires a live QEMU process with the right machine-type/ CPU-model to be started, to make the query. Requirement: libvirt needs to know if a specific CPU model can be used in the current host. See "Ensuring predictable set of guest features" above See "Querying host capabilities" below = Querying host capabilities = Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process. The set of available capabilities depend on: • Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options (e.g. in-kernel IRQ chip is required for some features). Current problem: libvirt uses the CPUID intruction directly and assumes that the presence of a feature in the host CPU means it can be enabled and exposed to the guest. This breaks when virtualization of a feature requires: • Additional hardware support (e.g. INVPCID); • Additional host kernel code (this applies to _all_ CPU features, that need to be reported as supported by GET_SUPPORTED_CPUID); • Additional QEMU-side code; • Specific configuration options (e.g. in-kernel IRQ chip). Challenge: QEMU doesn't have a generic capability-querying interface, and host capability querying depends on KVM to be initialized. Workaround: start a paused VM using the "host" CPU model, that has every single CPU feature supported by the host enabled by default, and query for the information about the CPU though QMP, using the QOM commands. Current solution: start a paused VM with no devices but with "host" CPU model and use QMP QOM commands to query for the enabled CPU features. Dependency: X86CPU feature properties == Future plans == It would be interesting to have a more generic capability-querying interface that doesn't require starting a whole machine with a live QEMU process. See also: -query-capabilities RFC series from Anthony Message-Id: <1332169763-30665-9-git-send-email-aliguori@us.ibm.com> -- Eduardo
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
Hi,
After a long time trying to figure out the proper modelling inside QEMU, I believe the plans are now clearer in QEMU, so it's time to coordinate more closely with libvirt to try to make use of the new stuff.
I tried to enumerate the libvirt requirements and current problems, and how we should be able to solve those problems using the X86CPU subclasses and properties, on the following wiki page:
http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvi...
= Ensuring predictable set of guest features =
Requirement: libvirt needs to ensure all features required on the command-line are present and exposed to the guest.
Current problem: libvirt doesn't use the "enforce" flag so it can't guarantee that a given feature will be actually exposed to the guest.
Solution: use the "enforce" flag on the "-cpu" option.
Definitely, we plan to start using "enforce" flag as soon as we have better CPU probing interface with QEMU. Since libvirt does not currently consult CPU specs with QEMU, some configurations in fact rely on QEMU dropping features it can't provide. Of course, that's bad for several reasons but we don't want such configurations to suddenly stop working. We want to first fix the CPU specs libvirt creates so that we know they will work with "enforce".
Limitation: no proper machine-friendly interface to report which features are missing.
Workaround: See "querying for host capabilities" below.
I doubt we will be ready to start using "enforce" before the machine friendly interface is available...
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
Solution: use QMP qom-list-types command.
Dependency: X86CPU subclasses. Limitation: needs a live QEMU process for the query.
No problem, we already run QEMU and use several QMP commands to probe its capabilities. And "qom-list-types" is actually one of them. To get the list of CPU models, we would just call { "execute": "qom-list-types", "arguments": { "implements": "X86CPU" } } right? What about other non-x86 architectures? Will we need to use different class name or is there a generic CPU class that could be used universally?
Solution: use QMP query-cpu-definitions command.
Limitation: needs a live QEMU process for the query.
IIUC, the result of this command will depend on machine type and we can't use -M none we currently use for probing, right?
== Future plans ==
It would be interesting to get rid of the requirement for a live QEMU process (with a complete machine being created) to be already running.
Hmm, so is this complete machine needed even for getting CPU models from qom-list-types or only for querying exact definitions using query-cpu-definitions command? Actually, what is query-cpu-definitions supposed to return? Currently it seems it's just the CPU model names rather than details about all CPU models. From the command name, one would expect to get more than just names.
= Getting information about CPU models =
Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user.
Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true.
Challenge: the resulting CPU features depend on lots of factors, including the machine-type.
Workaround: start a paused VM and query for the CPU device information after the CPU was created.
Solution: start a paused VM with no devices, but with the right machine-type and right CPU model. Use QMP QOM commands to query for CPU flags (especially the properties starting with the "f-" prefix).
Dependency: X86CPU feature properties ("f-*" properties). Limitation: requires a live QEMU process with the right machine-type/ CPU-model to be started, to make the query.
This would be very useful for ensuring the guest sees the exact same CPU after it's been migrated or restored from a stored state or a snapshot. Should we make sure the guest will always see the same CPU even after shutdown or is it ok if the guest CPU changes a bit on next boot, e.g., in case the host kernel was upgraded and is able to provide more features? However, probing several CPU definitions for compatibility with host/kernel/QEMU would be quite inefficient. Although I guess we should be able to limit doing so only in case a specific API (e.g., virConnectCompareCPU or virConnectBaselineCPU) is called, which would be acceptable, I think.
= Querying host capabilities =
Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process.
If we have machine friendly output of -cpu enforce, we may limit this need to just a few configuration types (the ones effectively requesting some kind of "host" CPU model). For other cases, libvirt could just try to start QEMU with the requested CPU definition and report usable error back rather than trying to pre-check the CPU. Jirka
On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
Hi,
After a long time trying to figure out the proper modelling inside QEMU, I believe the plans are now clearer in QEMU, so it's time to coordinate more closely with libvirt to try to make use of the new stuff.
I tried to enumerate the libvirt requirements and current problems, and how we should be able to solve those problems using the X86CPU subclasses and properties, on the following wiki page:
http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvi...
= Ensuring predictable set of guest features =
Requirement: libvirt needs to ensure all features required on the command-line are present and exposed to the guest.
Current problem: libvirt doesn't use the "enforce" flag so it can't guarantee that a given feature will be actually exposed to the guest.
Solution: use the "enforce" flag on the "-cpu" option.
Definitely, we plan to start using "enforce" flag as soon as we have better CPU probing interface with QEMU. Since libvirt does not currently consult CPU specs with QEMU, some configurations in fact rely on QEMU dropping features it can't provide. Of course, that's bad for several reasons but we don't want such configurations to suddenly stop working. We want to first fix the CPU specs libvirt creates so that we know they will work with "enforce".
Also: more important than fixing the CPU definitions from libvirt, is to ask QEMU for host capabilities and CPU model definitions. The whole point of this is to solve the CPU model data duplication/synchronization problems between libvirt and QEMU. Once you are able to query CPU model definitions on runtime, you don't even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU how each model looks like, and remove/add features from the command-line as necessary, so the resulting VM matches what the user asked for.
Limitation: no proper machine-friendly interface to report which features are missing.
Workaround: See "querying for host capabilities" below.
I doubt we will be ready to start using "enforce" before the machine friendly interface is available...
If you query for the "-cpu host" capabilities first and ensure all features from a CPU model is available, enforce is supposed to not fail. I understand that a machine-friendly error reporting for "enforce" would be very useful, but note that if "enforce" fails, it is probably already too late for libvirt, and that means that what libvirt thinks about host capabilities and CPU models is already incorrect. The main problem preventing us from making a machine-friendly interface is that "enforce" makes QEMU abort immediately, making us lose the main machine-friendly communication mechanism, that is QMP. (But I had an idea to solve that, look for "removed-features" below for a description).
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
Solution: use QMP qom-list-types command.
Dependency: X86CPU subclasses. Limitation: needs a live QEMU process for the query.
No problem, we already run QEMU and use several QMP commands to probe its capabilities. And "qom-list-types" is actually one of them. To get the list of CPU models, we would just call
{ "execute": "qom-list-types", "arguments": { "implements": "X86CPU" } }
right? What about other non-x86 architectures? Will we need to use different class name or is there a generic CPU class that could be used universally?
Actually I don't know much about the QMP command syntax and didn't test it a lot. But that's basically how I think it will look like. Except that instead of "X86CPU", the type name is "x86_64-cpu" (on qemu-system-x86_64) and "i386-cpu" (on qemu-system-i386). Maybe it is easier to simply use: implements: "cpu", abstract: false }. We may also end up with different CPU model classes for KVM and TCG, this is still under discussion. Another caveat: the CPU model class names will be longer than the names used on the "-cpu" command-line: something like "<model>-<arch>-cpu" or "<model>-kvm-<arch>-cpu".
Solution: use QMP query-cpu-definitions command.
Limitation: needs a live QEMU process for the query.
IIUC, the result of this command will depend on machine type and we can't use -M none we currently use for probing, right?
The class data don't depend on machine-types. But the resulting CPU objects may look different depending on machine type. In other words: you don't need -M to list CPU models, but you need -M if you want to know which features are going to be available on each model, exactly.
== Future plans ==
It would be interesting to get rid of the requirement for a live QEMU process (with a complete machine being created) to be already running.
Hmm, so is this complete machine needed even for getting CPU models from qom-list-types or only for querying exact definitions using query-cpu-definitions command?
Maybe "complete machine" isn't the right expression, here. What I mean is: AFAIK, it is not possible to get a QMP monitor without actually having a machine being created by QEMU (even if it is a machine that will never run). But answering your question: to query the CPU model list you don't need the right machine type or anything fancy. "qemu -nodefaults" should work. But to query the exact CPU definition (the item below), you need to specify the right machine-type.
Actually, what is query-cpu-definitions supposed to return? Currently it seems it's just the CPU model names rather than details about all CPU models. From the command name, one would expect to get more than just names.
query-cpu-definitions returns only the CPU model names, and a proposal to include more detailed CPUID information was rejected, IIRC. I believe we should deprecate query-cpu-definitions it in favor of the QOM commands.
= Getting information about CPU models =
Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user.
Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true.
Challenge: the resulting CPU features depend on lots of factors, including the machine-type.
Workaround: start a paused VM and query for the CPU device information after the CPU was created.
I just noticed another problem here, but this gave me an idea that would help solve the "enforce" error reporting problem: Problem: "qemu -machine <M> -cpu <model>" will create CPU objects where the CPU features are _already_ filtered based on host capabilities. * Using "enforce" wouldn't solve it, because then QEMU would abort, and QMP would be unavailable. Solution: we could have a CPU object property like "removed-features" that would have the list of features that were disabled because they are not supported by the host (and would make "enforce" fail). * This would solve the problem above and also be a machine-friendly way to check for possible "enforce" errors. * In other words: instead of "enforce", libvirt could use "check" instead of "enforce", and before unpausing the VM (or even starting migration), it should first check if the "removed-features" property is empty. Would that work for you?
Solution: start a paused VM with no devices, but with the right machine-type and right CPU model. Use QMP QOM commands to query for CPU flags (especially the properties starting with the "f-" prefix).
Dependency: X86CPU feature properties ("f-*" properties). Limitation: requires a live QEMU process with the right machine-type/ CPU-model to be started, to make the query.
This would be very useful for ensuring the guest sees the exact same CPU after it's been migrated or restored from a stored state or a snapshot. Should we make sure the guest will always see the same CPU even after shutdown or is it ok if the guest CPU changes a bit on next boot, e.g., in case the host kernel was upgraded and is able to provide more features?
It is up to you, it depends on what the libvirt API promises. But I always assumed that the same libvirt XML config would always create an equivalent machine (with not guest-visible differences), it didn't matter on which host it is running. Otherwise you would have problems like: * A VM being migratable to another host and suddenly it stops being migratable just because it was restarted. * A VM being migratable from A to B when started on host A, but not migratable from B to A if exactly the same VM XML is started on host B.
However, probing several CPU definitions for compatibility with host/kernel/QEMU would be quite inefficient. Although I guess we should be able to limit doing so only in case a specific API (e.g., virConnectCompareCPU or virConnectBaselineCPU) is called, which would be acceptable, I think.
I am not familiar with the libvirt code, but maybe the only cases where you need to do the probing are: * virConnectCompareCPU() * virConnectBaselineCPU() (I don't understand completely what it does) * Before starting a VM, when using match={minimu,exact,strict}. * When "host-model"/VIR_DOMAIN_XML_UPDATE_CPU is used * When checking if a given VM is migratable to a specific host (before actually starting migration). How is that implemented today? But I worry that at some point you may need to list detailed information about every CPU model, and this would require running QEMU once for each CPU model. Would that be a huge problem, or something we can work with? I quickly tested how quick QEMU can answer QMP queries: $ time sh -c '(echo "{\"execute\":\"qmp_capabilities\"}";echo "{\"execute\":\"qom-list-types\", \"arguments\": {}}";echo "{\"execute\":\"quit\"}"; ) | qemu-system-x86_64 -S -nodefaults -cpu SandyBridge,check -qmp stdio -nographic' [...] real 0m0.038s user 0m0.024s sys 0m0.012s Maybe this is reasonable enough? You probably just need to do that before starting a VM you never started before (for the cases where you want to choose the CPU model that would generate the shorter command-line). Also, note that the resulting CPU features for a given machine-type + CPU pair is supposed to never ever change on any future QEMU version, so libvirt should be able to safely cache those values even if the QEMU binary changes.
= Querying host capabilities =
Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process.
If we have machine friendly output of -cpu enforce, we may limit this need to just a few configuration types (the ones effectively requesting some kind of "host" CPU model). For other cases, libvirt could just try to start QEMU with the requested CPU definition and report usable error back rather than trying to pre-check the CPU.
True. PS: I will be out of office for 3 weeks, starting tomorrow, so I will probably not be able to continue this discussion after today, until I'm back on 25/March. -- Eduardo
On Fri, Mar 01, 2013 at 12:02:07 -0300, Eduardo Habkost wrote:
On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote:
Definitely, we plan to start using "enforce" flag as soon as we have better CPU probing interface with QEMU. Since libvirt does not currently consult CPU specs with QEMU, some configurations in fact rely on QEMU dropping features it can't provide. Of course, that's bad for several reasons but we don't want such configurations to suddenly stop working. We want to first fix the CPU specs libvirt creates so that we know they will work with "enforce".
Also: more important than fixing the CPU definitions from libvirt, is to ask QEMU for host capabilities and CPU model definitions. The whole point of this is to solve the CPU model data duplication/synchronization problems between libvirt and QEMU.
Once you are able to query CPU model definitions on runtime, you don't even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU how each model looks like, and remove/add features from the command-line as necessary, so the resulting VM matches what the user asked for.
Right, that's what I had in mind. By libvirt providing correct CPU definitions (probably better called the right -cpu command line option) I meant that we need to actually probe QEMU rather than making the definitions on our own from cpu_map.xml and host CPUID.
Limitation: no proper machine-friendly interface to report which features are missing.
Workaround: See "querying for host capabilities" below.
I doubt we will be ready to start using "enforce" before the machine friendly interface is available...
If you query for the "-cpu host" capabilities first and ensure all features from a CPU model is available, enforce is supposed to not fail.
I understand that a machine-friendly error reporting for "enforce" would be very useful, but note that if "enforce" fails, it is probably already too late for libvirt, and that means that what libvirt thinks about host capabilities and CPU models is already incorrect.
But we still need to know details about each CPU model so that we can choose the right one. I was trying to say that by the time we have this probing and can start using enforce, the machine friendly reporting could be available as well and the limitation will be gone.
== Future plans ==
It would be interesting to get rid of the requirement for a live QEMU process (with a complete machine being created) to be already running.
Hmm, so is this complete machine needed even for getting CPU models from qom-list-types or only for querying exact definitions using query-cpu-definitions command?
Maybe "complete machine" isn't the right expression, here. What I mean is: AFAIK, it is not possible to get a QMP monitor without actually having a machine being created by QEMU (even if it is a machine that will never run).
OK, I thought there was something special needed. We already start QEMU in such a way that we can communicate with it using QMP monitor. So this is just a question of using the right machine if we need to know the details about given CPU model.
= Getting information about CPU models =
Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user.
Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true.
Challenge: the resulting CPU features depend on lots of factors, including the machine-type.
Workaround: start a paused VM and query for the CPU device information after the CPU was created.
I just noticed another problem here, but this gave me an idea that would help solve the "enforce" error reporting problem:
Problem: "qemu -machine <M> -cpu <model>" will create CPU objects where the CPU features are _already_ filtered based on host capabilities.
Ah, it seems logical now that you mention it :-)
* Using "enforce" wouldn't solve it, because then QEMU would abort, and QMP would be unavailable.
Solution: we could have a CPU object property like "removed-features" that would have the list of features that were disabled because they are not supported by the host (and would make "enforce" fail).
* This would solve the problem above and also be a machine-friendly way to check for possible "enforce" errors.
* In other words: instead of "enforce", libvirt could use "check" instead of "enforce", and before unpausing the VM (or even starting migration), it should first check if the "removed-features" property is empty.
Would that work for you?
Yes, that seems like it could work. In fact, it seems much better than using enforce and trying to deal with aborted QEMU. Jirka
On Fri, Mar 01, 2013 at 11:56:00PM +0100, Jiri Denemark wrote: [...]
= Getting information about CPU models =
Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user.
Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true.
Challenge: the resulting CPU features depend on lots of factors, including the machine-type.
Workaround: start a paused VM and query for the CPU device information after the CPU was created.
I just noticed another problem here, but this gave me an idea that would help solve the "enforce" error reporting problem:
Problem: "qemu -machine <M> -cpu <model>" will create CPU objects where the CPU features are _already_ filtered based on host capabilities.
Ah, it seems logical now that you mention it :-)
* Using "enforce" wouldn't solve it, because then QEMU would abort, and QMP would be unavailable.
Solution: we could have a CPU object property like "removed-features" that would have the list of features that were disabled because they are not supported by the host (and would make "enforce" fail).
* This would solve the problem above and also be a machine-friendly way to check for possible "enforce" errors.
* In other words: instead of "enforce", libvirt could use "check" instead of "enforce", and before unpausing the VM (or even starting migration), it should first check if the "removed-features" property is empty.
Would that work for you?
Yes, that seems like it could work. In fact, it seems much better than using enforce and trying to deal with aborted QEMU.
To make the libvirt logic simpler, we could do this: have a "force" mode (in addition to check/enforce), that wouldn't filter any CPU feature based on host capabilities. This way libvirt wouldn't need any non-trivial logic to combine the "f-*" flags and "removed-features" to find out the CPU model details. Then libvirt would need to look only at "f-*" to find out the CPU model details at probing time (as long as "force" is used at probing time), and just make sure "removed-features" is empty before starting the VM (optionally parsing its value or checking the "f-*" property values, to find out which features are missing exactly). -- Eduardo
Am 01.03.2013 14:12, schrieb Jiri Denemark:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
query-cpu-definitions is the QMP command to retrieve values compatible with -cpu. And if libvirt is not using it, I really don't understand why the work of maintaining this crappy interface has been pushed onto us in the first place? There is no reuse between -cpu ? and QMP implementations so it's just extra work, there is no communicated or implemented way to extend the arch_query_cpu_definitions() implementation to become more usable for command line output implementation (e.g., associating a PVR value with the model name for ppc) and, while we're at it, it uses global functions plus a stub rather than a CPUState hook with a no-op default implementation in qom/cpu.c...
Solution: use QMP qom-list-types command.
Dependency: X86CPU subclasses. Limitation: needs a live QEMU process for the query.
No problem, we already run QEMU and use several QMP commands to probe its capabilities. And "qom-list-types" is actually one of them. To get the list of CPU models, we would just call
{ "execute": "qom-list-types", "arguments": { "implements": "X86CPU" } }
right?
Not quite, this would return abstract types as well, so you'd need to add "abstract": "false" or so. And you need to use the type name, not the struct name as argument, i.e. "i386-cpu" or "x86_64-cpu". Note: This does not always match the executable name since QOM names are supposed to be verbose (e.g., ppc64 vs. powerpc64-cpu) and some executables are misnamed (sh4 vs. superh-cpu). For x86 today this will return one type, either "i386-cpu" or "x86_64-cpu", that's why I have been pushing to implement model subclasses. There's still some controversial discussions about how this relates to KVM and TCG changing values of classes.
What about other non-x86 architectures?
For some other architectures like arm this will return the full list of available classes, but in "cortex-a9-arm-cpu" format, which is not guaranteed to be compatible with -cpu but rather with -device where already applicable.
Will we need to use different class name or is there a generic CPU class that could be used universally?
"cpu" would currently work as well, but the CPU refactorings are targetting to compile, e.g., arm-cpu and microblaze-cpu types into the same executable, so "cpu" may lead to undesired results in the future depending on what assumptions your code makes. Regards, Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
Am 01.03.2013 14:12, schrieb Jiri Denemark:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
query-cpu-definitions is the QMP command to retrieve values compatible with -cpu.
Yep, Jiri is wrong here actually. -cpu parsing is what we used to do. With latest libvirt + QEMU we do use query-cpu-definitions and have no "help parsing" code used at all. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Mar 01, 2013 at 06:34:31PM +0000, Daniel P. Berrange wrote:
On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
Am 01.03.2013 14:12, schrieb Jiri Denemark:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
query-cpu-definitions is the QMP command to retrieve values compatible with -cpu.
Yep, Jiri is wrong here actually. -cpu parsing is what we used to do. With latest libvirt + QEMU we do use query-cpu-definitions and have no "help parsing" code used at all.
I wrote the above, not Jiri. I will fix the wiki page to note that. -- Eduardo
On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
Am 01.03.2013 14:12, schrieb Jiri Denemark:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
query-cpu-definitions is the QMP command to retrieve values compatible with -cpu.
And if libvirt is not using it, I really don't understand why the work of maintaining this crappy interface has been pushed onto us in the first place? There is no reuse between -cpu ? and QMP implementations so it's just extra work, there is no communicated or implemented way to extend the arch_query_cpu_definitions() implementation to become more usable for command line output implementation (e.g., associating a PVR value with the model name for ppc) and, while we're at it, it uses global functions plus a stub rather than a CPUState hook with a no-op default implementation in qom/cpu.c...
I have the same questions you have. :-) But my main complaint about query-cpu-definitions is not about the implementation: it's that the interface was introduced without taking into account the requirements of libvirt regarding CPU features. It was found to be not appropriate for what libvirt needs[1], but somehow it got applied anyway. [1] http://article.gmane.org/gmane.comp.emulators.qemu/164772 -- Eduardo
On Fri, Mar 01, 2013 at 03:58:18PM -0300, Eduardo Habkost wrote:
On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote:
Am 01.03.2013 14:12, schrieb Jiri Denemark:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used with the "-cpu" option.
Current problem: libvirt relies on help output parsing for that.
query-cpu-definitions is the QMP command to retrieve values compatible with -cpu.
And if libvirt is not using it, I really don't understand why the work of maintaining this crappy interface has been pushed onto us in the first place? There is no reuse between -cpu ? and QMP implementations so it's just extra work, there is no communicated or implemented way to extend the arch_query_cpu_definitions() implementation to become more usable for command line output implementation (e.g., associating a PVR value with the model name for ppc) and, while we're at it, it uses global functions plus a stub rather than a CPUState hook with a no-op default implementation in qom/cpu.c...
I have the same questions you have. :-)
But my main complaint about query-cpu-definitions is not about the implementation: it's that the interface was introduced without taking into account the requirements of libvirt regarding CPU features. It was found to be not appropriate for what libvirt needs[1], but somehow it got applied anyway.
The requirement at the time we did this was just to have a straight adaptation of the stuff libvirt parsed from -help, into QMP format. The query-cpu-definitions command served that purpose fine. It was expected that some of this may be sub-optimal in the long term. This is fine - we can either extend the existing data, or introduce brand new commands to deal with the problems. The key thing was just to get 100% into the QMP world for capabilities at that time, and then iterate to improve things. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Querying host capabilities =
Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process.
The set of available capabilities depend on:
• Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options (e.g. in-kernel IRQ chip is required for some features).
Actually, one more thing. Can any of these requirements change while a host is up and QEMU is not upgraded? I believe, host CPU capabilities can only change when the host starts. Kernel capabilities are a bit less clear since I guess they could possibly change when kvm module is unloaded and loaded back with a different options. QEMU capabilities should only change when different version is installed. And the specific configuration options are the most unclear to me. The reason I'm asking is whether libvirt could run-time cache CPU definitions (including all model details) in the same way we currently cache QEMU capabilities, such as availability of specific QMP commands. Jirka
On Fri, Mar 01, 2013 at 02:28:37PM +0100, Jiri Denemark wrote:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Querying host capabilities =
Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process.
The set of available capabilities depend on:
• Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options (e.g. in-kernel IRQ chip is required for some features).
Actually, one more thing. Can any of these requirements change while a host is up and QEMU is not upgraded? I believe, host CPU capabilities can only change when the host starts. Kernel capabilities are a bit less clear since I guess they could possibly change when kvm module is unloaded and loaded back with a different options. QEMU capabilities should only change when different version is installed. And the specific configuration options are the most unclear to me. The reason I'm asking is whether libvirt could run-time cache CPU definitions (including all model details) in the same way we currently cache QEMU capabilities, such as availability of specific QMP commands.
That's a good question. Let's check each item so I don't forget any detail:
• Host CPU (hardware) capabilities;
This shouldn't change without a host reboot.
• Kernel capabilities (reported by GET_SUPPORTED_CPUID);
This may possibly change if the KVM module is unloaded and reloaded with different options, but... I guess we should simply require libvirtd to be restarted if any user does that?
• QEMU capabilities;
This shouldn't change as long as the QEMU binary doesn't change.
• Specific configuration options (e.g. in-kernel IRQ chip is required for some features).
This part seems tricky. Currently kernel-irqchip is probably the only option that affects which features are available, but what if other QEMU options affect the set of features too? I believe the answer is to rely on machine-types. I mean: if a new option that affects "-cpu host" and the set of available CPU features is created, there are two options: 1) Using the default value; 2) Setting the option explicitly. 1) If using the default value, the default will depend on machine-type, so libvirt should already consider machine-type as an option that affects available-CPU-features. 2) If using an explicit value, libvirt should use the explicit value only after being changed to take into account the fact that the option affects available-CPU-features. So, let's add one more item to the list. The set of available capabilities depend on: • Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options: • kernel-irqchip (currently affects tsc-deadline and x2apic, but may affect other features in the future) • machine-type (may affect any feature in the future) -- Eduardo
participants (4)
-
Andreas Färber -
Daniel P. Berrange -
Eduardo Habkost -
Jiri Denemark