Re: [libvirt] [RFC 00/28] s390x CPU models: exposing features
(CCing libvirt people) On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands: Example: 1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM. How exactly would this be done using the new commands?
Unfortunately, CPU models on s390x are special and we have to take care of: - A CPU like z13 looks differently in various environments (under different LPAR versions, under different z/VM versions, under different KVM versions, export regulation) - we have _a lot_ of feature variability. - We still have certain features that are not published but might be implemented/introduced in the future. As they are a theoretical part of a CPU already, we have to find a way to model these future changes. - We still have certain features that are already published, but not implemented. Implementation might be added in the future in KVM. - We heavily rely on KVM to tell us which features it can actually virtualize - user space queries like "STFL(e)" give no guarantees. - Certain "subfeatures" were introduced in one run. In practice, they are always around, but in theory, subfeatures could be dropped in the future. - Just because two CPU models have the same features doesn't mean they are equal - some internal numbers might be different. E.g. we won't allow running a z13 under a zBC12 just by turning off features. - We cannot blindly enable all possible features for a CPU generation, the IBC "Instruction Blocking Control" in KVM will try to block instructions introduced with certain features. So a CPU generation always has some maximum feature set that is guaranteed to work.
It all boils down to a specific released CPU to have. a) A fixed feature set that we expect it to be have on every hypervisor. b) A variable part that depends on the hypervisor and that could be extended in the future (adding not yet implemented features) that we always want to enable later on. c) A variable part that we want to enable only if requested - nested virtualization ("vsie") and assists are one example.
But, the fixed feature set is not really what we want to use as a default. It is just like a really minimum, stable base.
So we have a) A "stable" CPU model for each released CPU that will never change and maps to the minimum feature set we expect to be around on all hypervisors. e.g. "z13-base" or "z10EC.2-base". These are migration safe. b) A "default" CPU model for each released CPU, that can change between QEMU versions and that will always include the features we expect to be around in our currently supported environments and will contain only features we expect to be stable. E.g. nested virtualization will not be contained in these models. These models are not migration safe, e.g "z13" or "z10EC.2". The feature set can differ between QEMU versions. c) An internal "maximum" CPU model for each generation that tells us which features were supported as a maximum back when the hardware was released. This will not be exposed
To not have to replicate all CPU model changes ("new default fetaures") in libvirt, to not duplicate the logic about compatibility and the like, our approach tries to keep all the QEMU logic in libvirt and provide standardized interfaces for libvirt to e.g. baseline, compare. This allows libvirt to not have to care about any model names or feature names, it can just pass the data from interface to interface and report it to the user.
Also, libvirt might want to know what the "host" model looks like and convert a CPU model to a migration safe variant. For this reason, a QMP command is added that can create a migration safe variant of a variable CPU model, indicating only the delta changes done to a stable model.
So we have: a) "query-cpu-model-expansion" - tell us what the "host" or a migration unsafe model looks like. Either falling back to a stable model or completely exposing all properties. We are interested in stable models. b) "query-cpu-model-comparison" - tell us how two CPU models compare, indicating which properties were responsible for the decision. c) "query-cpu-model-baseline" - create a new model out of two models, taking a requested level of stability into account.
As we are aware that e.g. x86 has their own idea of a CPU model and their existing implementation in place, but are also looking into to ways to e.g. expand the "host" CPU model to a detailed representation, we designed the "expansion" interface to also allow that.
Comments are very welcome, but please always keep the restrictions and specialties in mind when suggesting some major design changes.
The header update will be replaced by a kvm-next header update as soon as the VSIE patches are upstream. The major KVM interface changes are already part of kvm-next.
The current state is available on git://github.com/cohuck/qemu on branch "cpumodel-s390x".
David Hildenbrand (26): s390x/cpumodel: "host" and "qemu" as CPU subclasses s390x/cpumodel: expose CPU class properties s390x/cpumodel: generate CPU feature group lists s390x/cpumodel: introduce CPU feature group definitions s390x/cpumodel: register defined CPU models as subclasses s390x/cpumodel: store the CPU model in the CPU instance s390x/cpumodel: expose features and feature groups as properties s390x/cpumodel: let the CPU model handle feature checks s390x/cpumodel: check and apply the CPU model s390x/sclp: factor out preparation of cpu entries s390x/sclp: introduce sclp feature blocks s390x/sclp: indicate sclp features s390x/sclp: propagate the ibc val(lowest and unblocked ibc) s390x/sclp: propagate the mha via sclp s390x/sclp: propagate hmfai update linux headers (CPU model) s390x/kvm: allow runtime-instrumentation for "none" machine s390x/kvm: implement CPU model support s390x/kvm: disable host model for existing compat machines s390x/kvm: let the CPU model control CMM(A) qmp: add QMP interface "query-cpu-model-expansion" qmp: add QMP interface "query-cpu-model-comparison" qmp: add QMP interface "query-cpu-model-baseline" s390x/cpumodel: implement QMP interface "query-cpu-model-expansion" s390x/cpumodel: implement QMP interface "query-cpu-model-comparison" s390x/cpumodel: implement QMP interface "query-cpu-model-baseline"
Michael Mueller (2): s390x/cpumodel: introduce CPU features s390x/cpumodel: generate CPU feature lists for CPU models
Makefile.target | 2 +- hw/s390x/s390-virtio-ccw.c | 5 + hw/s390x/s390-virtio.c | 6 +- hw/s390x/sclp.c | 35 +- include/hw/s390x/sclp.h | 17 +- include/sysemu/arch_init.h | 10 + linux-headers/asm-s390/kvm.h | 40 ++ qapi-schema.json | 184 ++++++ qmp-commands.hx | 18 + qmp.c | 22 + rules.mak | 1 + stubs/Makefile.objs | 3 + stubs/arch-query-cpu-model-baseline.c | 13 + stubs/arch-query-cpu-model-comparison.c | 12 + stubs/arch-query-cpu-model-expansion.c | 12 + target-s390x/Makefile.objs | 22 +- target-s390x/cpu-qom.h | 5 + target-s390x/cpu.c | 35 +- target-s390x/cpu.h | 5 + target-s390x/cpu_features.c | 376 +++++++++++ target-s390x/cpu_features.h | 302 +++++++++ target-s390x/cpu_models.c | 1055 +++++++++++++++++++++++++++++++ target-s390x/cpu_models.h | 113 ++++ target-s390x/gen-features.c | 587 +++++++++++++++++ target-s390x/helper.c | 29 +- target-s390x/kvm.c | 346 +++++++++- target-s390x/machine.c | 14 +- 27 files changed, 3203 insertions(+), 66 deletions(-) create mode 100644 stubs/arch-query-cpu-model-baseline.c create mode 100644 stubs/arch-query-cpu-model-comparison.c create mode 100644 stubs/arch-query-cpu-model-expansion.c create mode 100644 target-s390x/cpu_features.c create mode 100644 target-s390x/cpu_features.h create mode 100644 target-s390x/cpu_models.c create mode 100644 target-s390x/cpu_models.h create mode 100644 target-s390x/gen-features.c
-- 2.6.6
-- Eduardo
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands:
Example:
1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM.
How exactly would this be done using the new commands?
Hi Eduardo, thanks for having a look - highly appreciated that you actually map this to libvirt requirements! That would map to a compare operation between "host" and "Westmere,aes=on". Host could at that point already be expanded by libvirt. Doesn't matter at that point. If the result is "identica"l or "superset", it is runnable. If the result is "subset" or "incompatible", details about the responsible properties is indicated. (I actually took that idea from your patch for indicating runnability). Thanks! David
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands:
Example:
1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM.
How exactly would this be done using the new commands?
Hi Eduardo,
thanks for having a look - highly appreciated that you actually map this to libvirt requirements!
That would map to a compare operation between "host" and "Westmere,aes=on".
Host could at that point already be expanded by libvirt. Doesn't matter at that point.
If the result is "identica"l or "superset", it is runnable. If the result is "subset" or "incompatible", details about the responsible properties is indicated. (I actually took that idea from your patch for indicating runnability).
So, I have two worries about the proposal: 1) "query-cpu-model-expansion model=host" vs "query-host-cpu": I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running". For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported. If we don't mind adding multiple new QMP commands in this series, I don't see why not having a separate "query-host-cpu" command so the semantics of each command are very clear. Then query-cpu-definitions/query-cpu-model-expansion would be about "what you really get when using -cpu <model>", and "query-host-cpu" would be about "what this hosts supports running". 2) Requiring a running QEMU instance to run query-cpu-model-comparison With my previous query-host-cpu proposal, the task of comparing the configuration requested by the user with host capabilities can be done directly by libvirt. This way, no extra QEMU instance needs to be run before starting a VM. In my example above, if query-cpu-model-comparison were not required, libvirt could run "query-cpu-expansion model=host" or "query-host-cpu" once at boot, and just check if "Westmere" is runnable and "aes" is present in the host data. With query-cpu-model-comparison, QEMU needs to be run one more time before starting a VM. I don't know if requiring running a new QEMU instance just to check if a CPU config is runnable would be acceptable for libvirt. I will let the libvirt people answer that. I still like the query-cpu-model-comparison proposal, because it would allow us to extend the data returned by query-cpu-model-expansion without requiring libvirt changes. It may be the best log-term option, but probably only after we add a mechanism to allow the VM to be configured after running a few QMP commands, without re-running QEMU. -- Eduardo
On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands:
Example:
1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM.
How exactly would this be done using the new commands?
Hi Eduardo,
thanks for having a look - highly appreciated that you actually map this to libvirt requirements!
That would map to a compare operation between "host" and "Westmere,aes=on".
Host could at that point already be expanded by libvirt. Doesn't matter at that point.
If the result is "identica"l or "superset", it is runnable. If the result is "subset" or "incompatible", details about the responsible properties is indicated. (I actually took that idea from your patch for indicating runnability).
So, I have two worries about the proposal:
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running".
For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported.
What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86.
2) Requiring a running QEMU instance to run query-cpu-model-comparison
With my previous query-host-cpu proposal, the task of comparing the configuration requested by the user with host capabilities can be done directly by libvirt. This way, no extra QEMU instance needs to be run before starting a VM.
I think we can just easily get around this by not comparing a guest CPU to host (except for the explicit virConnectCompareCPU, which is not very useful in its current form anyway). Jirka
On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote: [...]
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running".
For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported.
What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86.
In that case, I think it makes sense to just extend query-cpu-definitions or use "query-cpu-model-expansion model=host" instead of a query-host-cpu command. Probably query-cpu-model-expansion is better than just extending query-cpu-definitions, because it would allow the expansion of extra CPU options, like "host,migratable=off". -- Eduardo
On Tue, Jun 21, 2016 at 18:22:30 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote: [...]
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running".
For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported.
What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86.
In that case, I think it makes sense to just extend query-cpu-definitions or use "query-cpu-model-expansion model=host" instead of a query-host-cpu command.
Probably query-cpu-model-expansion is better than just extending query-cpu-definitions, because it would allow the expansion of extra CPU options, like "host,migratable=off".
Yeah, this would be even better. Jirka
On Tue, Jun 21, 2016 at 18:22:30 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote: [...]
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running".
For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported.
What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86.
In that case, I think it makes sense to just extend query-cpu-definitions or use "query-cpu-model-expansion model=host" instead of a query-host-cpu command.
Probably query-cpu-model-expansion is better than just extending query-cpu-definitions, because it would allow the expansion of extra CPU options, like "host,migratable=off".
Yeah, this would be even better.
Jirka
Please be aware that we don't have anything like that on s390x, but I prepared for that requirement by being able to tell query-cpu-model-expansion how to expand (full, migratable, stable). Actually full and migratable looks the same on s390x. The plan for us is to rely on "stable" most of the time, full and migratable might be what you're looking for. David
On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands:
Example:
1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM.
How exactly would this be done using the new commands?
Hi Eduardo,
thanks for having a look - highly appreciated that you actually map this to libvirt requirements!
That would map to a compare operation between "host" and "Westmere,aes=on".
Host could at that point already be expanded by libvirt. Doesn't matter at that point.
If the result is "identica"l or "superset", it is runnable. If the result is "subset" or "incompatible", details about the responsible properties is indicated. (I actually took that idea from your patch for indicating runnability).
So, I have two worries about the proposal:
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running".
For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported.
What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86.
That's also what I had in mind first. Let me explain why they are not the same on s390x and why it is problematic (actually both types are not the same!): 1. Certain CPU features are only valid for LPAR guests, they can't be virtualized for KVM guests (remember that we always have at least one level of virtualization on s390x). So we will never be able to provide these features to a KVM guest, therefore we will never be able to completely mimic the real host model. 2. We have a whole lot of unpublished features. We would have to include them in the "real host model", but we can't. For the "host" model, this is not a problem, because we simply don't virtualize them. But the "real host model" would then vary between say QEMZ versions, which looks really strage, because in essance, QEMU/KVM looks like the wrong interface to query for a "real host model". 3. We don't have the kernel interfaces in place to query the "real host model". We can only query what we are able to virtualize. And that is indeed different compared to what I said previously. And as I can't see any use case for s390x, we most probably will never be able to support the interface you are suggesting here. Which is fine, if it makes sense for x86.
2) Requiring a running QEMU instance to run query-cpu-model-comparison
With my previous query-host-cpu proposal, the task of comparing the configuration requested by the user with host capabilities can be done directly by libvirt. This way, no extra QEMU instance needs to be run before starting a VM.
I think we can just easily get around this by not comparing a guest CPU to host (except for the explicit virConnectCompareCPU, which is not very useful in its current form anyway).
If there is some flexible way around that, great. But I think we (s390x) could life without this additional query. So your first concern is valid, and if you really need this information, you should probably really create a new interface. Your second concern should not be trivial for now. We don't want to optimize for total speed at that point. David
On Wed, Jun 22, 2016 at 08:51:40 +0200, David Hildenbrand wrote:
On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote: I still don't think we want to set in stone that "the result the guest sees when using -cpu host" is always the same as "what the host supports running".
For example: let's assume a given architecture have two features (A and B) that are both supported by the host but can never be enabled together. For actual "-cpu host" usage, QEMU would have to choose between enabling A and B. For querying host capabilities, we still want to let management software know that either A or B are supported.
What libvirt is really interested in is the guest CPU which would be used with -cpu host. This is actually what I thought query-host-cpu was all about. Perhaps because there's no difference for x86.
That's also what I had in mind first. Let me explain why they are not the same on s390x and why it is problematic (actually both types are not the same!):
1. Certain CPU features are only valid for LPAR guests, they can't be virtualized for KVM guests (remember that we always have at least one level of virtualization on s390x). So we will never be able to provide these features to a KVM guest, therefore we will never be able to completely mimic the real host model.
2. We have a whole lot of unpublished features. We would have to include them in the "real host model", but we can't. For the "host" model, this is not a problem, because we simply don't virtualize them. But the "real host model" would then vary between say QEMZ versions, which looks really strage, because in essance, QEMU/KVM looks like the wrong interface to query for a "real host model".
3. We don't have the kernel interfaces in place to query the "real host model". We can only query what we are able to virtualize. And that is indeed different compared to what I said previously.
And as I can't see any use case for s390x, we most probably will never be able to support the interface you are suggesting here. Which is fine, if it makes sense for x86.
Well, as I said I always thought about query-host-cpu as a way to get the CPU configuration QEMU would provide with -cpu host. Thanks to this discussion I realized its semantics is different and thus what I really need is actually the command you suggested, i.e., query-cpu-model-expansion.
2) Requiring a running QEMU instance to run query-cpu-model-comparison
With my previous query-host-cpu proposal, the task of comparing the configuration requested by the user with host capabilities can be done directly by libvirt. This way, no extra QEMU instance needs to be run before starting a VM.
I think we can just easily get around this by not comparing a guest CPU to host (except for the explicit virConnectCompareCPU, which is not very useful in its current form anyway).
If there is some flexible way around that, great. But I think we (s390x) could life without this additional query.
So if I understand correctly, you say you don't need the API to compare guest CPU to host CPU, right? If so, that's exactly what I said too. Jirka
2) Requiring a running QEMU instance to run query-cpu-model-comparison
With my previous query-host-cpu proposal, the task of comparing the configuration requested by the user with host capabilities can be done directly by libvirt. This way, no extra QEMU instance needs to be run before starting a VM.
I think we can just easily get around this by not comparing a guest CPU to host (except for the explicit virConnectCompareCPU, which is not very useful in its current form anyway).
If there is some flexible way around that, great. But I think we (s390x) could life without this additional query.
So if I understand correctly, you say you don't need the API to compare guest CPU to host CPU, right? If so, that's exactly what I said too.
I think the coffee didn't do its work already :) . I wanted to write that we can _with_ this additional query. Meaning the involved overhead would be ok - in my opinion for s390x. What we could do to avoid one compare operation would be: a) Expand the host model b) Expand the target model (because on s390x we could have migration unsafe model) c) Work with the runnability information returned via query-cpu-definitions But as we have to do b) either way on s390x, we can directly do a compare operation. (which makes implementation a lot simpler, because libvirt then doesn't have to deal with any feature/model names). David
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
I think the coffee didn't do its work already :) . I wanted to write that we can _with_ this additional query. Meaning the involved overhead would be ok - in my opinion for s390x.
What we could do to avoid one compare operation would be:
a) Expand the host model b) Expand the target model (because on s390x we could have migration unsafe model) c) Work with the runnability information returned via query-cpu-definitions
But as we have to do b) either way on s390x, we can directly do a compare operation. (which makes implementation a lot simpler, because libvirt then doesn't have to deal with any feature/model names).
But why do you even need to do any comparison? Isn't it possible to let QEMU do it when a domain starts? The thing is we should avoid doing completely different things on each architecture. Jirka
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
I think the coffee didn't do its work already :) . I wanted to write that we can _with_ this additional query. Meaning the involved overhead would be ok - in my opinion for s390x.
What we could do to avoid one compare operation would be:
a) Expand the host model b) Expand the target model (because on s390x we could have migration unsafe model) c) Work with the runnability information returned via query-cpu-definitions
But as we have to do b) either way on s390x, we can directly do a compare operation. (which makes implementation a lot simpler, because libvirt then doesn't have to deal with any feature/model names).
But why do you even need to do any comparison? Isn't it possible to let QEMU do it when a domain starts? The thing is we should avoid doing completely different things on each architecture.
Sure, QEMU will of course double check when starting the guest! So trying to start and failing is of course an option! So no check is needed if that is acceptable. David
On Wed, Jun 22, 2016 at 09:54:51 +0200, David Hildenbrand wrote:
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
I think the coffee didn't do its work already :) . I wanted to write that we can _with_ this additional query. Meaning the involved overhead would be ok - in my opinion for s390x.
What we could do to avoid one compare operation would be:
a) Expand the host model b) Expand the target model (because on s390x we could have migration unsafe model) c) Work with the runnability information returned via query-cpu-definitions
But as we have to do b) either way on s390x, we can directly do a compare operation. (which makes implementation a lot simpler, because libvirt then doesn't have to deal with any feature/model names).
But why do you even need to do any comparison? Isn't it possible to let QEMU do it when a domain starts? The thing is we should avoid doing completely different things on each architecture.
Sure, QEMU will of course double check when starting the guest! So trying to start and failing is of course an option! So no check is needed if that is acceptable.
Yeah, I think it's the safest and easiest option now. Jirka
On Wed, Jun 22, 2016 at 09:54:51 +0200, David Hildenbrand wrote:
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
I think the coffee didn't do its work already :) . I wanted to write that we can _with_ this additional query. Meaning the involved overhead would be ok - in my opinion for s390x.
What we could do to avoid one compare operation would be:
a) Expand the host model b) Expand the target model (because on s390x we could have migration unsafe model) c) Work with the runnability information returned via query-cpu-definitions
But as we have to do b) either way on s390x, we can directly do a compare operation. (which makes implementation a lot simpler, because libvirt then doesn't have to deal with any feature/model names).
But why do you even need to do any comparison? Isn't it possible to let QEMU do it when a domain starts? The thing is we should avoid doing completely different things on each architecture.
Sure, QEMU will of course double check when starting the guest! So trying to start and failing is of course an option! So no check is needed if that is acceptable.
Yeah, I think it's the safest and easiest option now.
Jirka
Alright then, this RFC already handles that properly, so that seems to be solved. The question now is if you guys see a fundamental problem in the way we want to handle CPU models. Especially a) Having flexible, not migration safe CPU models that can be expanded to migration safe models (using the expansion interface). b) Letting QEMU carry out the task of comparing and baselining to be used for e.g. for "virsh cpu-baseline" or "virsh cpu-compare". c) Indicating the host model as the expansion of "-cpu host", e.g. for "virsh capabilities" (which says "host" for now for us). Also, it will be good to know if the "expansion" interface with parameters "full" or "migratable" is really helpful to you or if I should drop that and you will come up with an own "query-host-cpu". We are also planning to implement the "query-cpu-definitions" runnability information in the future, because as you said, it might be a good way to quickly indicate runnable CPU models. But we are most likely not going to use it for e.g. comparing or baselining or detection of runnability of a more complex cpu model (having a lot of feature changes). Thanks! David
On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote:
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands:
Example:
1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM.
While this is what libvirt currently does, I don't think it's necessary to keep doing that... see below.
a) "query-cpu-model-expansion" - tell us what the "host" or a migration unsafe model looks like. Either falling back to a stable model or completely exposing all properties. We are interested in stable models. b) "query-cpu-model-comparison" - tell us how two CPU models compare, indicating which properties were responsible for the decision. c) "query-cpu-model-baseline" - create a new model out of two models, taking a requested level of stability into account.
This looks like it copies current libvirt APIs, which I think is not a very good idea. Both CPU compare and baseline APIs in libvirt will need to be changed (or rather new APIs with similar functionality added) to become useful. I think we should first focus on making guest CPU configuration work the way it should work. For that I think we need some higher level commands. Let me sum up what libvirt is doing (or will be doing) with guest CPUs... Libvirt supports three guest CPU configuration modes: - host-passthrough -- this is easy, it's just asking for "-cpu host" and no fancy commands are required. - host-model -- for this we need to know what CPU configuration we need to ask for to get as close to the host CPU as possible while being able to ask for the exact same CPU on a different host (after migration). And we need to be able to ask for this at probing time (when libvirtd starts rather than when starting a new domain) using "-machine none,accel=kvm:tcg", that is without actually creating any guest CPU. This is what Eduardo's query-host-cpu QMP command is useful for. In x86 world we could just query the guest CPU after running QEMU with "-cpu host", but that's unusable with "-machine none", which is why another way of getting this info is needed. - custom -- the XML specifies an exact guest CPU configuration. We don't really need to know if that exact CPU is runnable on the current host in advance, we can just try to start the domain, check if the guest CPU matches what we asked for, and we may kill QEMU if the CPU does not meet the specification. Of course, higher level management wants to know a set of host where it can run a given domain for scheduling purposes, but since they logically want to avoid tons of different CPU configs, they would just stick the CPU models predefined by QEMU. Thus giving them a way of checking what CPU models can be run on a given host with a given QEMU (using the unavailable features stuff for query-cpu-definitions usable with "-machine none") should be enough and it's even better than having to ask for every single CPU model, which is what they need to do now. There are configuration options which are somewhere between host-model and custom, but they don't bring more requirements in addition to what both of them needs. This was basically all about starting a new domain. When the domain successfully starts, we need to make sure the guest CPU does not change during save/restore or migration to another host. To achieve this, we need the same checking we need for custom mode, i.e., whether the guest CPU we got is what we asked for. In addition to this, we need a way to ask QEMU what the guest CPU looks like so that we can store it in the domain XML and ask for it during migration. I think all of this should be pretty much architecture agnostic. Of course the actual data would be quite different fro each architecture, but I believe the entry points could fit all. Or did I miss anything? Jirka
On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote:
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the following could be accomplished using the new commands:
Example:
1) User configures libvirt with: <cpu match='exact'> <model fallback='forbid'>Westmere</model> <feature policy='require' name='aes'/> </cpu> 2) libvirt will translate that to: "-cpu Westmere,+aes" or "-cpu Westmere,aes=on" 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in the current host, before trying to start the VM.
While this is what libvirt currently does, I don't think it's necessary to keep doing that... see below.
a) "query-cpu-model-expansion" - tell us what the "host" or a migration unsafe model looks like. Either falling back to a stable model or completely exposing all properties. We are interested in stable models. b) "query-cpu-model-comparison" - tell us how two CPU models compare, indicating which properties were responsible for the decision. c) "query-cpu-model-baseline" - create a new model out of two models, taking a requested level of stability into account.
This looks like it copies current libvirt APIs, which I think is not a very good idea. Both CPU compare and baseline APIs in libvirt will need to be changed (or rather new APIs with similar functionality added) to become useful. I think we should first focus on making guest CPU configuration work the way it should work. For that I think we need some higher level commands.
Hi Jiri, thanks for your reply! Please be aware that we (s390x) have no libvirt cpu model support in place. We are coming up with a solution that solves our problems, but instead of tagging all interfaces with "s390x" we want interfaces that everybody might use. I understand that x86 has some very advanced libvirt CPU model magic in place. You can still stick to what you have right now. It doesn't work for us. So we provide 3 basic functions (baseline, compare, expand) that can easily by added for s390x into the CPU model. So I don't see a problem introducing these operations in libvirt - it just works without any magic and knowledge about features and models in libvirt. In the end it keeps the CPU model where it belongs (QEMU) and allows us to just fit in fine into the libvirt CPU model world that was heavily inspired by x86 (an I can tell you, that is a very challenging task). But - of course - we have to do some additional queries on a QEMU instance. I don't think that this is a problem (at least not for s390x).
Let me sum up what libvirt is doing (or will be doing) with guest CPUs... Libvirt supports three guest CPU configuration modes:
One important thing to mention here is that all of these was inspired by x86. Some stuff doesn't even make sense on s390x ("mimic the real host"), as it would even in practice never be possible. See my reply to Eduards patchset I'll post later. We want to focus on the real use cases and I think they are - Making sure models ("Guest API") doesn't change during migration - Allowing the user to use a certain CPU model - Baselining/Comparing/Using the -cpu host model
- host-passthrough -- this is easy, it's just asking for "-cpu host" and no fancy commands are required.
This should map to expanding the host model in our case.
- host-model -- for this we need to know what CPU configuration we need to ask for to get as close to the host CPU as possible while being able to ask for the exact same CPU on a different host (after migration). And we need to be able to ask for this at probing time (when libvirtd starts rather than when starting a new domain) using "-machine none,accel=kvm:tcg", that is without actually creating any guest CPU. This is what Eduardo's query-host-cpu QMP command is useful for. In x86 world we could just query the guest CPU after running QEMU with "-cpu host", but that's unusable with "-machine none", which is why another way of getting this info is needed.
Again, we will be relying on "-cpu host" here, because the real host model will never be possible to be fordwarded on s390x. And what's the use case anyway? Does not make sense on s390x as far as I can tell. We want to know what we are able to virtualize, not build up some strange model that won't even be able to run.
- custom -- the XML specifies an exact guest CPU configuration. We don't really need to know if that exact CPU is runnable on the current host in advance, we can just try to start the domain, check if the guest CPU matches what we asked for, and we may kill QEMU if the CPU does not meet the specification. Of course, higher level management wants to know a set of host where it can run a given domain for scheduling purposes, but since they logically want to avoid tons of different CPU configs, they would just stick the CPU models predefined by QEMU. Thus giving them a way of checking what CPU models can be run on a given host with a given QEMU (using the unavailable features stuff for query-cpu-definitions usable with "-machine none") should be enough and it's even better than having to ask for every single CPU model, which is what they need to do now.
Which is simply necessary when having to care about a lot of feature variability. Please keep the requirements in our cover letter in place. It's not that simple when a) having to update the "flexible" models over time (e.g. adding stuff to "z13") b) nobody really using the "stable" models like "z13-base", but interested in "z13" + feature changes I agree that CPU models defined in QEMU are the place to go - which is why we keep our default here. And also keep in mind that I think your extension to query-cpu-definitions to detect runability makes total sense, it is just not enough for us having to care about changable "default" models. Let's say we report a "z13" via that interface as runnable? It doesn't mean a thing because that model could look different on e.g. another QEMU version we're migrating to. To check the real deal, we have to expand that model and then verify if it is runnable. In essence, runnability information contained in query-cpu-definitions only makes sense for migration-safe models. Of course, that could somehow be done using the "expansion" interface only in combination with "query-cpu-definitions". But it doesn't solve the other problems we're dealing with and just introduces some unnecessary complexity (for us!).
There are configuration options which are somewhere between host-model and custom, but they don't bring more requirements in addition to what both of them needs.
This was basically all about starting a new domain. When the domain successfully starts, we need to make sure the guest CPU does not change during save/restore or migration to another host. To achieve this, we need the same checking we need for custom mode, i.e., whether the guest CPU we got is what we asked for. In addition to this, we need a way to ask QEMU what the guest CPU looks like so that we can store it in the domain XML and ask for it during migration.
Exactly, that's where our "expansion" interface comes into play. If the user asks for a "z13", we will expand it to a "z13-base,feata=on,featb=on" ... So we can guarantee that we are always dealing with a migration safe variant.
I think all of this should be pretty much architecture agnostic. Of course the actual data would be quite different fro each architecture, but I believe the entry points could fit all. Or did I miss anything?
These are also the use cases I looked into, with the addition of being able to baseline and compare CPU models (including the "host" model). Thanks again for having a look! David
On Tue, Jun 21, 2016 at 10:56:28PM +0200, Jiri Denemark wrote: [...]
Let me sum up what libvirt is doing (or will be doing) with guest CPUs... Libvirt supports three guest CPU configuration modes:
- host-passthrough -- this is easy, it's just asking for "-cpu host" and no fancy commands are required.
- host-model -- for this we need to know what CPU configuration we need to ask for to get as close to the host CPU as possible while being able to ask for the exact same CPU on a different host (after migration). And we need to be able to ask for this at probing time (when libvirtd starts rather than when starting a new domain) using "-machine none,accel=kvm:tcg", that is without actually creating any guest CPU. This is what Eduardo's query-host-cpu QMP command is useful for. In x86 world we could just query the guest CPU after running QEMU with "-cpu host", but that's unusable with "-machine none", which is why another way of getting this info is needed.
Question: KVM supports x2apic (and enables it by default) on any host CPU, because it is all emulated by KVM. Should "host-model" include x2apic on all hosts, or only when the host CPU has it? ("-cpu host" does include it). Including x2apic sounds more useful, but it doesn't match the "get as close to the host CPU as possible" part of your description. -- Eduardo
Question: KVM supports x2apic (and enables it by default) on any host CPU, because it is all emulated by KVM. Should "host-model" include x2apic on all hosts, or only when the host CPU has it? ("-cpu host" does include it).
Including x2apic sounds more useful, but it doesn't match the "get as close to the host CPU as possible" part of your description.
From a s390x point of view, our "host" model is about indicating the maximum
possible configuration we are able to virtualize, not strictly limiting it only to the real host features. We also have a feature called "sthyi", emulated by KVM, which might not be around for the real "host". We will include that in "host", because otherwise things seem to get really complicated with no obvious benefit (to me :) ). Not sure how you want to handle that. David
participants (3)
-
David Hildenbrand -
Eduardo Habkost -
Jiri Denemark