Are there any further comments, especially on patches 23-25, introducing new QOM interfaces? Also, if anybody is planning to look into this, please speak up :) Otherwise it might make sense to put this onto the next KVM call agenda. David

This is our second attempt to implement CPU models for s390x. We realized that we also want to have features exposed via the CPU model. While doing that we realized that we want to have a better interface for libvirt.

Unfortunately, CPU models on s390x are special and we have to take care of: - A CPU like z13 looks differently in various environments (under different LPAR versions, under different z/VM versions, under different KVM versions, export regulation) - we have _a lot_ of feature variability. - We still have certain features that are not published but might be implemented/introduced in the future. As they are a theoretical part of a CPU already, we have to find a way to model these future changes. - We still have certain features that are already published, but not implemented. Implementation might be added in the future in KVM. - We heavily rely on KVM to tell us which features it can actually virtualize - user space queries like "STFL(e)" give no guarantees. - Certain "subfeatures" were introduced in one run. In practice, they are always around, but in theory, subfeatures could be dropped in the future. - Just because two CPU models have the same features doesn't mean they are equal - some internal numbers might be different. E.g. we won't allow running a z13 under a zBC12 just by turning off features. - We cannot blindly enable all possible features for a CPU generation, the IBC "Instruction Blocking Control" in KVM will try to block instructions introduced with certain features. So a CPU generation always has some maximum feature set that is guaranteed to work.

It all boils down to a specific released CPU to have. a) A fixed feature set that we expect it to be have on every hypervisor. b) A variable part that depends on the hypervisor and that could be extended in the future (adding not yet implemented features) that we always want to enable later on. c) A variable part that we want to enable only if requested - nested virtualization ("vsie") and assists are one example.

But, the fixed feature set is not really what we want to use as a default. It is just like a really minimum, stable base.

So we have a) A "stable" CPU model for each released CPU that will never change and maps to the minimum feature set we expect to be around on all hypervisors. e.g. "z13-base" or "z10EC.2-base". These are migration safe. b) A "default" CPU model for each released CPU, that can change between QEMU versions and that will always include the features we expect to be around in our currently supported environments and will contain only features we expect to be stable. E.g. nested virtualization will not be contained in these models. These models are not migration safe, e.g "z13" or "z10EC.2". The feature set can differ between QEMU versions. c) An internal "maximum" CPU model for each generation that tells us which features were supported as a maximum back when the hardware was released. This will not be exposed

To not have to replicate all CPU model changes ("new default fetaures") in libvirt, to not duplicate the logic about compatibility and the like, our approach tries to keep all the QEMU logic in libvirt and provide standardized interfaces for libvirt to e.g. baseline, compare. This allows libvirt to not have to care about any model names or feature names, it can just pass the data from interface to interface and report it to the user.

Also, libvirt might want to know what the "host" model looks like and convert a CPU model to a migration safe variant. For this reason, a QMP command is added that can create a migration safe variant of a variable CPU model, indicating only the delta changes done to a stable model.

So we have: a) "query-cpu-model-expansion" - tell us what the "host" or a migration unsafe model looks like. Either falling back to a stable model or completely exposing all properties. We are interested in stable models. b) "query-cpu-model-comparison" - tell us how two CPU models compare, indicating which properties were responsible for the decision. c) "query-cpu-model-baseline" - create a new model out of two models, taking a requested level of stability into account.

As we are aware that e.g. x86 has their own idea of a CPU model and their existing implementation in place, but are also looking into to ways to e.g. expand the "host" CPU model to a detailed representation, we designed the "expansion" interface to also allow that.

Comments are very welcome, but please always keep the restrictions and specialties in mind when suggesting some major design changes.

The header update will be replaced by a kvm-next header update as soon as the VSIE patches are upstream. The major KVM interface changes are already part of kvm-next.

The current state is available on git://github.com/cohuck/qemu on branch "cpumodel-s390x".

David Hildenbrand (26): s390x/cpumodel: "host" and "qemu" as CPU subclasses s390x/cpumodel: expose CPU class properties s390x/cpumodel: generate CPU feature group lists s390x/cpumodel: introduce CPU feature group definitions s390x/cpumodel: register defined CPU models as subclasses s390x/cpumodel: store the CPU model in the CPU instance s390x/cpumodel: expose features and feature groups as properties s390x/cpumodel: let the CPU model handle feature checks s390x/cpumodel: check and apply the CPU model s390x/sclp: factor out preparation of cpu entries s390x/sclp: introduce sclp feature blocks s390x/sclp: indicate sclp features s390x/sclp: propagate the ibc val(lowest and unblocked ibc) s390x/sclp: propagate the mha via sclp s390x/sclp: propagate hmfai update linux headers (CPU model) s390x/kvm: allow runtime-instrumentation for "none" machine s390x/kvm: implement CPU model support s390x/kvm: disable host model for existing compat machines s390x/kvm: let the CPU model control CMM(A) qmp: add QMP interface "query-cpu-model-expansion" qmp: add QMP interface "query-cpu-model-comparison" qmp: add QMP interface "query-cpu-model-baseline" s390x/cpumodel: implement QMP interface "query-cpu-model-expansion" s390x/cpumodel: implement QMP interface "query-cpu-model-comparison" s390x/cpumodel: implement QMP interface "query-cpu-model-baseline"

Michael Mueller (2): s390x/cpumodel: introduce CPU features s390x/cpumodel: generate CPU feature lists for CPU models

Makefile.target | 2 +- hw/s390x/s390-virtio-ccw.c | 5 + hw/s390x/s390-virtio.c | 6 +- hw/s390x/sclp.c | 35 +- include/hw/s390x/sclp.h | 17 +- include/sysemu/arch_init.h | 10 + linux-headers/asm-s390/kvm.h | 40 ++ qapi-schema.json | 184 ++++++ qmp-commands.hx | 18 + qmp.c | 22 + rules.mak | 1 + stubs/Makefile.objs | 3 + stubs/arch-query-cpu-model-baseline.c | 13 + stubs/arch-query-cpu-model-comparison.c | 12 + stubs/arch-query-cpu-model-expansion.c | 12 + target-s390x/Makefile.objs | 22 +- target-s390x/cpu-qom.h | 5 + target-s390x/cpu.c | 35 +- target-s390x/cpu.h | 5 + target-s390x/cpu_features.c | 376 +++++++++++ target-s390x/cpu_features.h | 302 +++++++++ target-s390x/cpu_models.c | 1055 +++++++++++++++++++++++++++++++ target-s390x/cpu_models.h | 113 ++++ target-s390x/gen-features.c | 587 +++++++++++++++++ target-s390x/helper.c | 29 +- target-s390x/kvm.c | 346 +++++++++- target-s390x/machine.c | 14 +- 27 files changed, 3203 insertions(+), 66 deletions(-) create mode 100644 stubs/arch-query-cpu-model-baseline.c create mode 100644 stubs/arch-query-cpu-model-comparison.c create mode 100644 stubs/arch-query-cpu-model-expansion.c create mode 100644 target-s390x/cpu_features.c create mode 100644 target-s390x/cpu_features.h create mode 100644 target-s390x/cpu_models.c create mode 100644 target-s390x/cpu_models.h create mode 100644 target-s390x/gen-features.c