11:44 a.m.
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
This is our second attempt to implement CPU models for s390x. We
realized
that we also want to have features exposed via the CPU model. While doing
that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the
following could be accomplished using the new commands:
Example:
1) User configures libvirt with:
<cpu match='exact'>
<model fallback='forbid'>Westmere</model>
<feature policy='require' name='aes'/>
</cpu>
2) libvirt will translate that to:
"-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
the current host, before trying to start the VM.
How exactly would this be done using the new commands?
Unfortunately, CPU models on s390x are special and we have to take care of:
- A CPU like z13 looks differently in various environments (under different
LPAR versions, under different z/VM versions, under different KVM
versions, export regulation) - we have _a lot_ of feature variability.
- We still have certain features that are not published but might be
implemented/introduced in the future. As they are a theoretical part
of a CPU already, we have to find a way to model these future changes.
- We still have certain features that are already published, but not
implemented. Implementation might be added in the future in KVM.
- We heavily rely on KVM to tell us which features it can actually
virtualize - user space queries like "STFL(e)" give no guarantees.
- Certain "subfeatures" were introduced in one run. In practice, they are
always around, but in theory, subfeatures could be dropped in the future.
- Just because two CPU models have the same features doesn't mean they
are equal - some internal numbers might be different. E.g. we won't allow
running a z13 under a zBC12 just by turning off features.
- We cannot blindly enable all possible features for a CPU generation,
the IBC "Instruction Blocking Control" in KVM will try to block
instructions introduced with certain features. So a CPU generation always
has some maximum feature set that is guaranteed to work.
It all boils down to a specific released CPU to have.
a) A fixed feature set that we expect it to be have on every hypervisor.
b) A variable part that depends on the hypervisor and that could be
extended in the future (adding not yet implemented features) that we
always want to enable later on.
c) A variable part that we want to enable only if requested - nested
virtualization ("vsie") and assists are one example.
But, the fixed feature set is not really what we want to use as a default.
It is just like a really minimum, stable base.
So we have
a) A "stable" CPU model for each released CPU that will never change and
maps to the minimum feature set we expect to be around on all
hypervisors. e.g. "z13-base" or "z10EC.2-base". These are
migration
safe.
b) A "default" CPU model for each released CPU, that can change between
QEMU versions and that will always include the features we expect to
be around in our currently supported environments and will contain only
features we expect to be stable. E.g. nested virtualization will not be
contained in these models. These models are not migration safe, e.g
"z13" or "z10EC.2". The feature set can differ between QEMU
versions.
c) An internal "maximum" CPU model for each generation that tells us which
features were supported as a maximum back when the hardware was
released. This will not be exposed
To not have to replicate all CPU model changes ("new default fetaures") in
libvirt, to not duplicate the logic about compatibility and the like,
our approach tries to keep all the QEMU logic in libvirt and provide
standardized interfaces for libvirt to e.g. baseline, compare. This
allows libvirt to not have to care about any model names or feature names,
it can just pass the data from interface to interface and report it to
the user.
Also, libvirt might want to know what the "host" model looks like and
convert a CPU model to a migration safe variant. For this reason, a QMP
command is added that can create a migration safe variant of a variable
CPU model, indicating only the delta changes done to a stable model.
So we have:
a) "query-cpu-model-expansion" - tell us what the "host" or a
migration
unsafe model looks like. Either falling back to a stable model or
completely exposing all properties. We are interested in stable models.
b) "query-cpu-model-comparison" - tell us how two CPU models compare,
indicating which properties were responsible for the decision.
c) "query-cpu-model-baseline" - create a new model out of two models,
taking a requested level of stability into account.
As we are aware that e.g. x86 has their own idea of a CPU model and their
existing implementation in place, but are also looking into to ways to e.g.
expand the "host" CPU model to a detailed representation, we designed the
"expansion" interface to also allow that.
Comments are very welcome, but please always keep the restrictions and
specialties in mind when suggesting some major design changes.
The header update will be replaced by a kvm-next header update as soon as
the VSIE patches are upstream. The major KVM interface changes are already
part of kvm-next.
The current state is available on git://github.com/cohuck/qemu on branch
"cpumodel-s390x".
David Hildenbrand (26):
s390x/cpumodel: "host" and "qemu" as CPU subclasses
s390x/cpumodel: expose CPU class properties
s390x/cpumodel: generate CPU feature group lists
s390x/cpumodel: introduce CPU feature group definitions
s390x/cpumodel: register defined CPU models as subclasses
s390x/cpumodel: store the CPU model in the CPU instance
s390x/cpumodel: expose features and feature groups as properties
s390x/cpumodel: let the CPU model handle feature checks
s390x/cpumodel: check and apply the CPU model
s390x/sclp: factor out preparation of cpu entries
s390x/sclp: introduce sclp feature blocks
s390x/sclp: indicate sclp features
s390x/sclp: propagate the ibc val(lowest and unblocked ibc)
s390x/sclp: propagate the mha via sclp
s390x/sclp: propagate hmfai
update linux headers (CPU model)
s390x/kvm: allow runtime-instrumentation for "none" machine
s390x/kvm: implement CPU model support
s390x/kvm: disable host model for existing compat machines
s390x/kvm: let the CPU model control CMM(A)
qmp: add QMP interface "query-cpu-model-expansion"
qmp: add QMP interface "query-cpu-model-comparison"
qmp: add QMP interface "query-cpu-model-baseline"
s390x/cpumodel: implement QMP interface "query-cpu-model-expansion"
s390x/cpumodel: implement QMP interface "query-cpu-model-comparison"
s390x/cpumodel: implement QMP interface "query-cpu-model-baseline"
Michael Mueller (2):
s390x/cpumodel: introduce CPU features
s390x/cpumodel: generate CPU feature lists for CPU models
Makefile.target | 2 +-
hw/s390x/s390-virtio-ccw.c | 5 +
hw/s390x/s390-virtio.c | 6 +-
hw/s390x/sclp.c | 35 +-
include/hw/s390x/sclp.h | 17 +-
include/sysemu/arch_init.h | 10 +
linux-headers/asm-s390/kvm.h | 40 ++
qapi-schema.json | 184 ++++++
qmp-commands.hx | 18 +
qmp.c | 22 +
rules.mak | 1 +
stubs/Makefile.objs | 3 +
stubs/arch-query-cpu-model-baseline.c | 13 +
stubs/arch-query-cpu-model-comparison.c | 12 +
stubs/arch-query-cpu-model-expansion.c | 12 +
target-s390x/Makefile.objs | 22 +-
target-s390x/cpu-qom.h | 5 +
target-s390x/cpu.c | 35 +-
target-s390x/cpu.h | 5 +
target-s390x/cpu_features.c | 376 +++++++++++
target-s390x/cpu_features.h | 302 +++++++++
target-s390x/cpu_models.c | 1055 +++++++++++++++++++++++++++++++
target-s390x/cpu_models.h | 113 ++++
target-s390x/gen-features.c | 587 +++++++++++++++++
target-s390x/helper.c | 29 +-
target-s390x/kvm.c | 346 +++++++++-
target-s390x/machine.c | 14 +-
27 files changed, 3203 insertions(+), 66 deletions(-)
create mode 100644 stubs/arch-query-cpu-model-baseline.c
create mode 100644 stubs/arch-query-cpu-model-comparison.c
create mode 100644 stubs/arch-query-cpu-model-expansion.c
create mode 100644 target-s390x/cpu_features.c
create mode 100644 target-s390x/cpu_features.h
create mode 100644 target-s390x/cpu_models.c
create mode 100644 target-s390x/cpu_models.h
create mode 100644 target-s390x/gen-features.c
--
2.6.6
--
Eduardo
12:01 p.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> This is our second attempt to implement CPU models for s390x. We realized
> that we also want to have features exposed via the CPU model. While doing
> that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the
following could be accomplished using the new commands:
Example:
1) User configures libvirt with:
<cpu match='exact'>
<model fallback='forbid'>Westmere</model>
<feature policy='require' name='aes'/>
</cpu>
2) libvirt will translate that to:
"-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
the current host, before trying to start the VM.
How exactly would this be done using the new commands?
Hi Eduardo,
thanks for having a look - highly appreciated that you actually map this
to libvirt requirements!
That would map to a compare operation between "host" and
"Westmere,aes=on".
Host could at that point already be expanded by libvirt. Doesn't matter at that
point.
If the result is "identica"l or "superset", it is runnable. If the
result is
"subset" or "incompatible", details about the responsible properties
is
indicated. (I actually took that idea from your patch for indicating
runnability).
Thanks!
David
3:33 p.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
> (CCing libvirt people)
>
> On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> > This is our second attempt to implement CPU models for s390x. We realized
> > that we also want to have features exposed via the CPU model. While doing
> > that we realized that we want to have a better interface for libvirt.
>
> Before getting into the details, I would like to clarify how the
> following could be accomplished using the new commands:
>
> Example:
>
> 1) User configures libvirt with:
> <cpu match='exact'>
> <model fallback='forbid'>Westmere</model>
> <feature policy='require' name='aes'/>
> </cpu>
> 2) libvirt will translate that to:
> "-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
> 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
> the current host, before trying to start the VM.
>
> How exactly would this be done using the new commands?
Hi Eduardo,
thanks for having a look - highly appreciated that you actually map this
to libvirt requirements!
That would map to a compare operation between "host" and
"Westmere,aes=on".
Host could at that point already be expanded by libvirt. Doesn't matter at that
point.
If the result is "identica"l or "superset", it is runnable. If the
result is
"subset" or "incompatible", details about the responsible properties
is
indicated. (I actually took that idea from your patch for indicating
runnability).
So, I have two worries about the proposal:
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the
guest sees when using -cpu host" is always the same as "what the
host supports running".
For example: let's assume a given architecture have two features
(A and B) that are both supported by the host but can never be
enabled together. For actual "-cpu host" usage, QEMU would have
to choose between enabling A and B. For querying host
capabilities, we still want to let management software know that
either A or B are supported.
If we don't mind adding multiple new QMP commands in this series,
I don't see why not having a separate "query-host-cpu" command so
the semantics of each command are very clear. Then
query-cpu-definitions/query-cpu-model-expansion would be about
"what you really get when using -cpu <model>", and
"query-host-cpu" would be about "what this hosts supports
running".
2) Requiring a running QEMU instance to run
query-cpu-model-comparison
With my previous query-host-cpu proposal, the task of comparing
the configuration requested by the user with host capabilities
can be done directly by libvirt. This way, no extra QEMU instance
needs to be run before starting a VM.
In my example above, if query-cpu-model-comparison were not
required, libvirt could run "query-cpu-expansion model=host" or
"query-host-cpu" once at boot, and just check if "Westmere" is
runnable and "aes" is present in the host data. With
query-cpu-model-comparison, QEMU needs to be run one more time
before starting a VM.
I don't know if requiring running a new QEMU instance just to
check if a CPU config is runnable would be acceptable for
libvirt. I will let the libvirt people answer that.
I still like the query-cpu-model-comparison proposal, because it
would allow us to extend the data returned by
query-cpu-model-expansion without requiring libvirt changes. It
may be the best log-term option, but probably only after we add a
mechanism to allow the VM to be configured after running a few
QMP commands, without re-running QEMU.
--
Eduardo
4:09 p.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
> > (CCing libvirt people)
> >
> > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> > > This is our second attempt to implement CPU models for s390x. We realized
> > > that we also want to have features exposed via the CPU model. While doing
> > > that we realized that we want to have a better interface for libvirt.
> >
> > Before getting into the details, I would like to clarify how the
> > following could be accomplished using the new commands:
> >
> > Example:
> >
> > 1) User configures libvirt with:
> > <cpu match='exact'>
> > <model fallback='forbid'>Westmere</model>
> > <feature policy='require' name='aes'/>
> > </cpu>
> > 2) libvirt will translate that to:
> > "-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
> > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
> > the current host, before trying to start the VM.
> >
> > How exactly would this be done using the new commands?
>
> Hi Eduardo,
>
> thanks for having a look - highly appreciated that you actually map this
> to libvirt requirements!
>
> That would map to a compare operation between "host" and
"Westmere,aes=on".
>
> Host could at that point already be expanded by libvirt. Doesn't matter at that
> point.
>
> If the result is "identica"l or "superset", it is runnable. If
the result is
> "subset" or "incompatible", details about the responsible
properties is
> indicated. (I actually took that idea from your patch for indicating
> runnability).
So, I have two worries about the proposal:
1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
I still don't think we want to set in stone that "the result the
guest sees when using -cpu host" is always the same as "what the
host supports running".
For example: let's assume a given architecture have two features
(A and B) that are both supported by the host but can never be
enabled together. For actual "-cpu host" usage, QEMU would have
to choose between enabling A and B. For querying host
capabilities, we still want to let management software know that
either A or B are supported.
What libvirt is really interested in is the guest CPU which would be
used with -cpu host. This is actually what I thought query-host-cpu was
all about. Perhaps because there's no difference for x86.
2) Requiring a running QEMU instance to run
query-cpu-model-comparison
With my previous query-host-cpu proposal, the task of comparing
the configuration requested by the user with host capabilities
can be done directly by libvirt. This way, no extra QEMU instance
needs to be run before starting a VM.
I think we can just easily get around this by not comparing a guest CPU
to host (except for the explicit virConnectCompareCPU, which is not very
useful in its current form anyway).
Jirka
4:22 p.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote:
[...]
> 1) "query-cpu-model-expansion model=host" vs
"query-host-cpu":
>
> I still don't think we want to set in stone that "the result the
> guest sees when using -cpu host" is always the same as "what the
> host supports running".
>
> For example: let's assume a given architecture have two features
> (A and B) that are both supported by the host but can never be
> enabled together. For actual "-cpu host" usage, QEMU would have
> to choose between enabling A and B. For querying host
> capabilities, we still want to let management software know that
> either A or B are supported.
What libvirt is really interested in is the guest CPU which would be
used with -cpu host. This is actually what I thought query-host-cpu was
all about. Perhaps because there's no difference for x86.
In that case, I think it makes sense to just extend
query-cpu-definitions or use "query-cpu-model-expansion
model=host" instead of a query-host-cpu command.
Probably query-cpu-model-expansion is better than just extending
query-cpu-definitions, because it would allow the expansion of
extra CPU options, like "host,migratable=off".
--
Eduardo
2:11 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 18:22:30 -0300, Eduardo Habkost wrote:
On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote:
[...]
> > 1) "query-cpu-model-expansion model=host" vs
"query-host-cpu":
> >
> > I still don't think we want to set in stone that "the result the
> > guest sees when using -cpu host" is always the same as "what the
> > host supports running".
> >
> > For example: let's assume a given architecture have two features
> > (A and B) that are both supported by the host but can never be
> > enabled together. For actual "-cpu host" usage, QEMU would have
> > to choose between enabling A and B. For querying host
> > capabilities, we still want to let management software know that
> > either A or B are supported.
>
> What libvirt is really interested in is the guest CPU which would be
> used with -cpu host. This is actually what I thought query-host-cpu was
> all about. Perhaps because there's no difference for x86.
In that case, I think it makes sense to just extend
query-cpu-definitions or use "query-cpu-model-expansion
model=host" instead of a query-host-cpu command.
Probably query-cpu-model-expansion is better than just extending
query-cpu-definitions, because it would allow the expansion of
extra CPU options, like "host,migratable=off".
Yeah, this would be even better.
Jirka
2:14 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 18:22:30 -0300, Eduardo Habkost wrote:
> On Tue, Jun 21, 2016 at 11:09:49PM +0200, Jiri Denemark wrote:
> [...]
> > > 1) "query-cpu-model-expansion model=host" vs
"query-host-cpu":
> > >
> > > I still don't think we want to set in stone that "the result the
> > > guest sees when using -cpu host" is always the same as "what
the
> > > host supports running".
> > >
> > > For example: let's assume a given architecture have two features
> > > (A and B) that are both supported by the host but can never be
> > > enabled together. For actual "-cpu host" usage, QEMU would have
> > > to choose between enabling A and B. For querying host
> > > capabilities, we still want to let management software know that
> > > either A or B are supported.
> >
> > What libvirt is really interested in is the guest CPU which would be
> > used with -cpu host. This is actually what I thought query-host-cpu was
> > all about. Perhaps because there's no difference for x86.
>
> In that case, I think it makes sense to just extend
> query-cpu-definitions or use "query-cpu-model-expansion
> model=host" instead of a query-host-cpu command.
>
> Probably query-cpu-model-expansion is better than just extending
> query-cpu-definitions, because it would allow the expansion of
> extra CPU options, like "host,migratable=off".
Yeah, this would be even better.
Jirka
Please be aware that we don't have anything like that on s390x, but
I prepared for that requirement by being able to tell
query-cpu-model-expansion how to expand (full, migratable, stable).
Actually full and migratable looks the same on s390x.
The plan for us is to rely on "stable" most of the time, full and migratable
might be what you're looking for.
David
1:51 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote:
> On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
> > > (CCing libvirt people)
> > >
> > > On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> > > > This is our second attempt to implement CPU models for s390x. We
realized
> > > > that we also want to have features exposed via the CPU model. While
doing
> > > > that we realized that we want to have a better interface for libvirt.
> > >
> > > Before getting into the details, I would like to clarify how the
> > > following could be accomplished using the new commands:
> > >
> > > Example:
> > >
> > > 1) User configures libvirt with:
> > > <cpu match='exact'>
> > > <model fallback='forbid'>Westmere</model>
> > > <feature policy='require' name='aes'/>
> > > </cpu>
> > > 2) libvirt will translate that to:
> > > "-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
> > > 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
> > > the current host, before trying to start the VM.
> > >
> > > How exactly would this be done using the new commands?
> >
> > Hi Eduardo,
> >
> > thanks for having a look - highly appreciated that you actually map this
> > to libvirt requirements!
> >
> > That would map to a compare operation between "host" and
"Westmere,aes=on".
> >
> > Host could at that point already be expanded by libvirt. Doesn't matter at
that
> > point.
> >
> > If the result is "identica"l or "superset", it is runnable.
If the result is
> > "subset" or "incompatible", details about the responsible
properties is
> > indicated. (I actually took that idea from your patch for indicating
> > runnability).
>
> So, I have two worries about the proposal:
>
>
> 1) "query-cpu-model-expansion model=host" vs "query-host-cpu":
>
> I still don't think we want to set in stone that "the result the
> guest sees when using -cpu host" is always the same as "what the
> host supports running".
>
> For example: let's assume a given architecture have two features
> (A and B) that are both supported by the host but can never be
> enabled together. For actual "-cpu host" usage, QEMU would have
> to choose between enabling A and B. For querying host
> capabilities, we still want to let management software know that
> either A or B are supported.
What libvirt is really interested in is the guest CPU which would be
used with -cpu host. This is actually what I thought query-host-cpu was
all about. Perhaps because there's no difference for x86.
That's also what I had in mind first. Let me explain why they are not the same
on s390x and why it is problematic (actually both types are not the same!):
1. Certain CPU features are only valid for LPAR guests, they can't be
virtualized for KVM guests (remember that we always have at least one level of
virtualization on s390x). So we will never be able to provide these features to
a KVM guest, therefore we will never be able to completely mimic the real host
model.
2. We have a whole lot of unpublished features. We would have to include them in
the "real host model", but we can't. For the "host" model, this is
not a
problem, because we simply don't virtualize them. But the "real host model"
would then vary between say QEMZ versions, which looks really strage, because
in essance, QEMU/KVM looks like the wrong interface to query for a "real host
model".
3. We don't have the kernel interfaces in place to query the "real host
model".
We can only query what we are able to virtualize. And that is indeed different
compared to what I said previously.
And as I can't see any use case for s390x, we most probably will never be able
to support the interface you are suggesting here. Which is fine, if it makes
sense for x86.
> 2) Requiring a running QEMU instance to run
> query-cpu-model-comparison
>
> With my previous query-host-cpu proposal, the task of comparing
> the configuration requested by the user with host capabilities
> can be done directly by libvirt. This way, no extra QEMU instance
> needs to be run before starting a VM.
I think we can just easily get around this by not comparing a guest CPU
to host (except for the explicit virConnectCompareCPU, which is not very
useful in its current form anyway).
If there is some flexible way around that, great. But I think we (s390x) could
life without this additional query.
So your first concern is valid, and if you really need this information, you
should probably really create a new interface. Your second concern should not
be trivial for now. We don't want to optimize for total speed at that point.
David
2:26 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 08:51:40 +0200, David Hildenbrand wrote:
> On Tue, Jun 21, 2016 at 17:33:09 -0300, Eduardo Habkost wrote:
> > On Tue, Jun 21, 2016 at 07:01:44PM +0200, David Hildenbrand wrote:
> > I still don't think we want to set in stone that "the result the
> > guest sees when using -cpu host" is always the same as "what the
> > host supports running".
> >
> > For example: let's assume a given architecture have two features
> > (A and B) that are both supported by the host but can never be
> > enabled together. For actual "-cpu host" usage, QEMU would have
> > to choose between enabling A and B. For querying host
> > capabilities, we still want to let management software know that
> > either A or B are supported.
>
> What libvirt is really interested in is the guest CPU which would be
> used with -cpu host. This is actually what I thought query-host-cpu was
> all about. Perhaps because there's no difference for x86.
That's also what I had in mind first. Let me explain why they are not the same
on s390x and why it is problematic (actually both types are not the same!):
1. Certain CPU features are only valid for LPAR guests, they can't be
virtualized for KVM guests (remember that we always have at least one level of
virtualization on s390x). So we will never be able to provide these features to
a KVM guest, therefore we will never be able to completely mimic the real host
model.
2. We have a whole lot of unpublished features. We would have to include them in
the "real host model", but we can't. For the "host" model, this
is not a
problem, because we simply don't virtualize them. But the "real host
model"
would then vary between say QEMZ versions, which looks really strage, because
in essance, QEMU/KVM looks like the wrong interface to query for a "real host
model".
3. We don't have the kernel interfaces in place to query the "real host
model".
We can only query what we are able to virtualize. And that is indeed different
compared to what I said previously.
And as I can't see any use case for s390x, we most probably will never be able
to support the interface you are suggesting here. Which is fine, if it makes
sense for x86.
Well, as I said I always thought about query-host-cpu as a way to get
the CPU configuration QEMU would provide with -cpu host. Thanks to this
discussion I realized its semantics is different and thus what I really
need is actually the command you suggested, i.e.,
query-cpu-model-expansion.
> > 2) Requiring a running QEMU instance to run
> > query-cpu-model-comparison
> >
> > With my previous query-host-cpu proposal, the task of comparing
> > the configuration requested by the user with host capabilities
> > can be done directly by libvirt. This way, no extra QEMU instance
> > needs to be run before starting a VM.
>
> I think we can just easily get around this by not comparing a guest CPU
> to host (except for the explicit virConnectCompareCPU, which is not very
> useful in its current form anyway).
If there is some flexible way around that, great. But I think we (s390x) could
life without this additional query.
So if I understand correctly, you say you don't need the API to compare
guest CPU to host CPU, right? If so, that's exactly what I said too.
Jirka
2:34 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
> > > 2) Requiring a running QEMU instance to run
> > > query-cpu-model-comparison
> > >
> > > With my previous query-host-cpu proposal, the task of comparing
> > > the configuration requested by the user with host capabilities
> > > can be done directly by libvirt. This way, no extra QEMU instance
> > > needs to be run before starting a VM.
> >
> > I think we can just easily get around this by not comparing a guest CPU
> > to host (except for the explicit virConnectCompareCPU, which is not very
> > useful in its current form anyway).
>
> If there is some flexible way around that, great. But I think we (s390x) could
> life without this additional query.
So if I understand correctly, you say you don't need the API to compare
guest CPU to host CPU, right? If so, that's exactly what I said too.
I think the coffee didn't do its work already :) . I wanted to write that we can
_with_ this additional query. Meaning the involved overhead would be ok - in my
opinion for s390x.
What we could do to avoid one compare operation would be:
a) Expand the host model
b) Expand the target model (because on s390x we could have migration unsafe
model)
c) Work with the runnability information returned via query-cpu-definitions
But as we have to do b) either way on s390x, we can directly do a compare
operation. (which makes implementation a lot simpler, because libvirt then
doesn't have to deal with any feature/model names).
David
2:53 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
I think the coffee didn't do its work already :) . I wanted to
write that we can
_with_ this additional query. Meaning the involved overhead would be ok - in my
opinion for s390x.
What we could do to avoid one compare operation would be:
a) Expand the host model
b) Expand the target model (because on s390x we could have migration unsafe
model)
c) Work with the runnability information returned via query-cpu-definitions
But as we have to do b) either way on s390x, we can directly do a compare
operation. (which makes implementation a lot simpler, because libvirt then
doesn't have to deal with any feature/model names).
But why do you even need to do any comparison? Isn't it possible to let
QEMU do it when a domain starts? The thing is we should avoid doing
completely different things on each architecture.
Jirka
2:54 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
> I think the coffee didn't do its work already :) . I wanted to write that we
can
> _with_ this additional query. Meaning the involved overhead would be ok - in my
> opinion for s390x.
>
> What we could do to avoid one compare operation would be:
>
> a) Expand the host model
> b) Expand the target model (because on s390x we could have migration unsafe
> model)
> c) Work with the runnability information returned via query-cpu-definitions
>
> But as we have to do b) either way on s390x, we can directly do a compare
> operation. (which makes implementation a lot simpler, because libvirt then
> doesn't have to deal with any feature/model names).
But why do you even need to do any comparison? Isn't it possible to let
QEMU do it when a domain starts? The thing is we should avoid doing
completely different things on each architecture.
Sure, QEMU will of course double check when starting the guest! So trying to
start and failing is of course an option! So no check is needed if that is
acceptable.
David
3:05 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 09:54:51 +0200, David Hildenbrand wrote:
> On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand
wrote:
> > I think the coffee didn't do its work already :) . I wanted to write that
we can
> > _with_ this additional query. Meaning the involved overhead would be ok - in
my
> > opinion for s390x.
> >
> > What we could do to avoid one compare operation would be:
> >
> > a) Expand the host model
> > b) Expand the target model (because on s390x we could have migration unsafe
> > model)
> > c) Work with the runnability information returned via query-cpu-definitions
> >
> > But as we have to do b) either way on s390x, we can directly do a compare
> > operation. (which makes implementation a lot simpler, because libvirt then
> > doesn't have to deal with any feature/model names).
>
> But why do you even need to do any comparison? Isn't it possible to let
> QEMU do it when a domain starts? The thing is we should avoid doing
> completely different things on each architecture.
>
Sure, QEMU will of course double check when starting the guest! So trying to
start and failing is of course an option! So no check is needed if that is
acceptable.
Yeah, I think it's the safest and easiest option now.
Jirka
1:41 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Wed, Jun 22, 2016 at 09:54:51 +0200, David Hildenbrand wrote:
> > On Wed, Jun 22, 2016 at 09:34:49 +0200, David Hildenbrand wrote:
> > > I think the coffee didn't do its work already :) . I wanted to write
that we can
> > > _with_ this additional query. Meaning the involved overhead would be ok -
in my
> > > opinion for s390x.
> > >
> > > What we could do to avoid one compare operation would be:
> > >
> > > a) Expand the host model
> > > b) Expand the target model (because on s390x we could have migration
unsafe
> > > model)
> > > c) Work with the runnability information returned via
query-cpu-definitions
> > >
> > > But as we have to do b) either way on s390x, we can directly do a compare
> > > operation. (which makes implementation a lot simpler, because libvirt
then
> > > doesn't have to deal with any feature/model names).
> >
> > But why do you even need to do any comparison? Isn't it possible to let
> > QEMU do it when a domain starts? The thing is we should avoid doing
> > completely different things on each architecture.
> >
>
> Sure, QEMU will of course double check when starting the guest! So trying to
> start and failing is of course an option! So no check is needed if that is
> acceptable.
Yeah, I think it's the safest and easiest option now.
Jirka
Alright then, this RFC already handles that properly, so that seems to be
solved. The question now is if you guys see a fundamental problem in the way we
want to handle CPU models.
Especially
a) Having flexible, not migration safe CPU models that can be expanded to
migration safe models (using the expansion interface).
b) Letting QEMU carry out the task of comparing and baselining to be used
for e.g. for "virsh cpu-baseline" or "virsh cpu-compare".
c) Indicating the host model as the expansion of "-cpu host", e.g. for
"virsh
capabilities" (which says "host" for now for us).
Also, it will be good to know if the "expansion" interface with parameters
"full" or "migratable" is really helpful to you or if I should drop
that and
you will come up with an own "query-host-cpu".
We are also planning to implement the "query-cpu-definitions" runnability
information in the future, because as you said, it might be a good way to
quickly indicate runnable CPU models. But we are most likely not going to use it
for e.g. comparing or baselining or detection of runnability of a more complex
cpu model (having a lot of feature changes).
Thanks!
David
3:56 p.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote:
(CCing libvirt people)
On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> This is our second attempt to implement CPU models for s390x. We realized
> that we also want to have features exposed via the CPU model. While doing
> that we realized that we want to have a better interface for libvirt.
Before getting into the details, I would like to clarify how the
following could be accomplished using the new commands:
Example:
1) User configures libvirt with:
<cpu match='exact'>
<model fallback='forbid'>Westmere</model>
<feature policy='require' name='aes'/>
</cpu>
2) libvirt will translate that to:
"-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
the current host, before trying to start the VM.
While this is what libvirt currently does, I don't think it's necessary
to keep doing that... see below.
> a) "query-cpu-model-expansion" - tell us what the
"host" or a migration
> unsafe model looks like. Either falling back to a stable model or
> completely exposing all properties. We are interested in stable models.
> b) "query-cpu-model-comparison" - tell us how two CPU models compare,
> indicating which properties were responsible for the decision.
> c) "query-cpu-model-baseline" - create a new model out of two models,
> taking a requested level of stability into account.
This looks like it copies current libvirt APIs, which I think is not a
very good idea. Both CPU compare and baseline APIs in libvirt will need
to be changed (or rather new APIs with similar functionality added) to
become useful. I think we should first focus on making guest CPU
configuration work the way it should work. For that I think we need some
higher level commands.
Let me sum up what libvirt is doing (or will be doing) with guest
CPUs... Libvirt supports three guest CPU configuration modes:
- host-passthrough -- this is easy, it's just asking for "-cpu host" and
no fancy commands are required.
- host-model -- for this we need to know what CPU configuration we need
to ask for to get as close to the host CPU as possible while being
able to ask for the exact same CPU on a different host (after
migration). And we need to be able to ask for this at probing time
(when libvirtd starts rather than when starting a new domain) using
"-machine none,accel=kvm:tcg", that is without actually creating any
guest CPU. This is what Eduardo's query-host-cpu QMP command is useful
for. In x86 world we could just query the guest CPU after running QEMU
with "-cpu host", but that's unusable with "-machine none",
which is
why another way of getting this info is needed.
- custom -- the XML specifies an exact guest CPU configuration. We don't
really need to know if that exact CPU is runnable on the current host
in advance, we can just try to start the domain, check if the guest
CPU matches what we asked for, and we may kill QEMU if the CPU does
not meet the specification. Of course, higher level management wants
to know a set of host where it can run a given domain for scheduling
purposes, but since they logically want to avoid tons of different CPU
configs, they would just stick the CPU models predefined by QEMU. Thus
giving them a way of checking what CPU models can be run on a given
host with a given QEMU (using the unavailable features stuff for
query-cpu-definitions usable with "-machine none") should be enough
and it's even better than having to ask for every single CPU model,
which is what they need to do now.
There are configuration options which are somewhere between host-model
and custom, but they don't bring more requirements in addition to what
both of them needs.
This was basically all about starting a new domain. When the domain
successfully starts, we need to make sure the guest CPU does not change
during save/restore or migration to another host. To achieve this, we
need the same checking we need for custom mode, i.e., whether the guest
CPU we got is what we asked for. In addition to this, we need a way to
ask QEMU what the guest CPU looks like so that we can store it in the
domain XML and ask for it during migration.
I think all of this should be pretty much architecture agnostic. Of
course the actual data would be quite different fro each architecture,
but I believe the entry points could fit all. Or did I miss anything?
Jirka
1:43 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 13:44:31 -0300, Eduardo Habkost wrote:
> (CCing libvirt people)
>
> On Tue, Jun 21, 2016 at 03:02:05PM +0200, David Hildenbrand wrote:
> > This is our second attempt to implement CPU models for s390x. We realized
> > that we also want to have features exposed via the CPU model. While doing
> > that we realized that we want to have a better interface for libvirt.
>
> Before getting into the details, I would like to clarify how the
> following could be accomplished using the new commands:
>
> Example:
>
> 1) User configures libvirt with:
> <cpu match='exact'>
> <model fallback='forbid'>Westmere</model>
> <feature policy='require' name='aes'/>
> </cpu>
> 2) libvirt will translate that to:
> "-cpu Westmere,+aes" or "-cpu Westmere,aes=on"
> 3) libvirt wants to know if "-cpu Westmere,aes=on" is usable in
> the current host, before trying to start the VM.
While this is what libvirt currently does, I don't think it's necessary
to keep doing that... see below.
> > a) "query-cpu-model-expansion" - tell us what the "host" or
a migration
> > unsafe model looks like. Either falling back to a stable model or
> > completely exposing all properties. We are interested in stable models.
> > b) "query-cpu-model-comparison" - tell us how two CPU models
compare,
> > indicating which properties were responsible for the decision.
> > c) "query-cpu-model-baseline" - create a new model out of two
models,
> > taking a requested level of stability into account.
This looks like it copies current libvirt APIs, which I think is not a
very good idea. Both CPU compare and baseline APIs in libvirt will need
to be changed (or rather new APIs with similar functionality added) to
become useful. I think we should first focus on making guest CPU
configuration work the way it should work. For that I think we need some
higher level commands.
Hi Jiri,
thanks for your reply!
Please be aware that we (s390x) have no libvirt cpu model support in place. We
are coming up with a solution that solves our problems, but instead of tagging
all interfaces with "s390x" we want interfaces that everybody might use.
I understand that x86 has some very advanced libvirt CPU model magic in place.
You can still stick to what you have right now. It doesn't work for us.
So we provide 3 basic functions (baseline, compare, expand) that can easily by
added for s390x into the CPU model. So I don't see a problem introducing
these operations in libvirt - it just works without any magic and knowledge
about features and models in libvirt.
In the end it keeps the CPU model where it belongs (QEMU) and allows us to
just fit in fine into the libvirt CPU model world that was heavily inspired
by x86 (an I can tell you, that is a very challenging task).
But - of course - we have to do some additional queries on a QEMU instance. I
don't think that this is a problem (at least not for s390x).
Let me sum up what libvirt is doing (or will be doing) with guest
CPUs... Libvirt supports three guest CPU configuration modes:
One important thing to mention here is that all of these was inspired by x86.
Some stuff doesn't even make sense on s390x ("mimic the real host"), as
it would even in practice never be possible. See my reply to Eduards patchset
I'll post later.
We want to focus on the real use cases and I think they are
- Making sure models ("Guest API") doesn't change during migration
- Allowing the user to use a certain CPU model
- Baselining/Comparing/Using the -cpu host model
- host-passthrough -- this is easy, it's just asking for "-cpu host" and
no fancy commands are required.
This should map to expanding the host model in our case.
- host-model -- for this we need to know what CPU configuration we need
to ask for to get as close to the host CPU as possible while being
able to ask for the exact same CPU on a different host (after
migration). And we need to be able to ask for this at probing time
(when libvirtd starts rather than when starting a new domain) using
"-machine none,accel=kvm:tcg", that is without actually creating any
guest CPU. This is what Eduardo's query-host-cpu QMP command is useful
for. In x86 world we could just query the guest CPU after running QEMU
with "-cpu host", but that's unusable with "-machine none",
which is
why another way of getting this info is needed.
Again, we will be relying on "-cpu host" here, because the real host model
will never be possible to be fordwarded on s390x. And what's the use case
anyway? Does not make sense on s390x as far as I can tell.
We want to know what we are able to virtualize, not build up some strange
model that won't even be able to run.
- custom -- the XML specifies an exact guest CPU configuration. We don't
really need to know if that exact CPU is runnable on the current host
in advance, we can just try to start the domain, check if the guest
CPU matches what we asked for, and we may kill QEMU if the CPU does
not meet the specification. Of course, higher level management wants
to know a set of host where it can run a given domain for scheduling
purposes, but since they logically want to avoid tons of different CPU
configs, they would just stick the CPU models predefined by QEMU. Thus
giving them a way of checking what CPU models can be run on a given
host with a given QEMU (using the unavailable features stuff for
query-cpu-definitions usable with "-machine none") should be enough
and it's even better than having to ask for every single CPU model,
which is what they need to do now.
Which is simply necessary when having to care about a lot of feature
variability. Please keep the requirements in our cover letter in place.
It's not that simple when
a) having to update the "flexible" models over time (e.g. adding stuff to
"z13")
b) nobody really using the "stable" models like "z13-base", but
interested in
"z13" + feature changes
I agree that CPU models defined in QEMU are the place to go - which is why we
keep our default here.
And also keep in mind that I think your extension to query-cpu-definitions to
detect runability makes total sense, it is just not enough for us having to care
about changable "default" models.
Let's say we report a "z13" via that interface as runnable? It doesn't
mean a thing because that model could look different on e.g. another QEMU
version we're migrating to. To check the real deal, we have to expand that model
and then verify if it is runnable. In essence, runnability information
contained in query-cpu-definitions only makes sense for migration-safe models.
Of course, that could somehow be done using the "expansion" interface only
in combination with "query-cpu-definitions". But it doesn't solve the other
problems we're dealing with and just introduces some unnecessary complexity
(for us!).
There are configuration options which are somewhere between host-model
and custom, but they don't bring more requirements in addition to what
both of them needs.
This was basically all about starting a new domain. When the domain
successfully starts, we need to make sure the guest CPU does not change
during save/restore or migration to another host. To achieve this, we
need the same checking we need for custom mode, i.e., whether the guest
CPU we got is what we asked for. In addition to this, we need a way to
ask QEMU what the guest CPU looks like so that we can store it in the
domain XML and ask for it during migration.
Exactly, that's where our "expansion" interface comes into play.
If the user asks for a "z13", we will expand it to a
"z13-base,feata=on,featb=on" ...
So we can guarantee that we are always dealing with a migration safe variant.
I think all of this should be pretty much architecture agnostic. Of
course the actual data would be quite different fro each architecture,
but I believe the entry points could fit all. Or did I miss anything?
These are also the use cases I looked into, with the addition of being able to
baseline and compare CPU models (including the "host" model).
Thanks again for having a look!
David
1:21 p.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
On Tue, Jun 21, 2016 at 10:56:28PM +0200, Jiri Denemark wrote:
[...]
Let me sum up what libvirt is doing (or will be doing) with guest
CPUs... Libvirt supports three guest CPU configuration modes:
- host-passthrough -- this is easy, it's just asking for "-cpu host" and
no fancy commands are required.
- host-model -- for this we need to know what CPU configuration we need
to ask for to get as close to the host CPU as possible while being
able to ask for the exact same CPU on a different host (after
migration). And we need to be able to ask for this at probing time
(when libvirtd starts rather than when starting a new domain) using
"-machine none,accel=kvm:tcg", that is without actually creating any
guest CPU. This is what Eduardo's query-host-cpu QMP command is useful
for. In x86 world we could just query the guest CPU after running QEMU
with "-cpu host", but that's unusable with "-machine none",
which is
why another way of getting this info is needed.
Question: KVM supports x2apic (and enables it by default) on any
host CPU, because it is all emulated by KVM. Should "host-model"
include x2apic on all hosts, or only when the host CPU has it?
("-cpu host" does include it).
Including x2apic sounds more useful, but it doesn't match the
"get as close to the host CPU as possible" part of your
description.
--
Eduardo
1:33 a.m.
New subject: [libvirt] [RFC 00/28] s390x CPU models: exposing features
Question: KVM supports x2apic (and enables it by default) on any
host CPU, because it is all emulated by KVM. Should "host-model"
include x2apic on all hosts, or only when the host CPU has it?
("-cpu host" does include it).
Including x2apic sounds more useful, but it doesn't match the
"get as close to the host CPU as possible" part of your
description.
From a s390x point of view, our "host" model is about
indicating the maximum
possible configuration we are able to virtualize, not
strictly limiting it only
to the real host features.
We also have a feature called "sthyi", emulated by KVM, which might not be
around for the real "host". We will include that in "host", because
otherwise
things seem to get really complicated with no obvious benefit (to me :) ).
Not sure how you want to handle that.
David
3074
days inactive
3076
days old
17 comments
3 participants
participants (3)
-
David Hildenbrand -
Eduardo Habkost -
Jiri Denemark