Re: [libvirt] [RFC] qemu: Redesigning guest CPU configuration

On 18.06.2015 15:41, Daniel P. Berrange wrote: > On Wed, Jun 17, 2015 at 05:37:42PM +0200, Jiri Denemark wrote: >> Hi all (and sorry for the long email), >> >> The current way QEMU driver handles guest CPU configuration is not >> ideal. We detect host CPU capabilities only by querying the CPU and we >> don't check with QEMU what features it supports. We don't check QEMU's >> definitions of CPU models, which may be different from libvirt's >> definitions. All this results in several issues: >> >> - guest CPU may change druing migration, save/restore >> - libvirt may ask for a CPU which QEMU cannot provide; the guest will >> see a slightly different CPU but libvirt client won't know about it >> - libvirt may come up with a CPU that doesn't make sense and which won't >> work for a guest (the guest may even crash) >> >> Although usually everything just works, it is very fragile. > > A third issue is that if there is no <cpu> in the guest config, we > just delegate CPU choice to QEMU and then ignore any CPU checks when > migrating. If libvirt owns the full CPU config, we'd probably want > to also decide the default ourselves, so that we will always be able > todo migrate CPU checks. > >> Since we want to fix all these issues, we need to: >> - guarantee stable guest ABI (a single domain XML should always results >> in the same guest ABI). Once a domain is started, its CPU definition >> should never change (unless someone changes the XML, of course, >> similar to, e.g. PCI addresses). However, there are a few exceptions: >> - host-passthrough CPU mode will always result in "-cpu host" >> - host-model CPU mode should recompute the CPU model on every start, >> but the CPU must not change during migration >> - always make sure QEMU provides the CPU we asked for. Starting a domain >> should fail in case QEMU cannot provide exactly the CPU we asked for. >> - provide usable host-model mode and custom mode with minimum match. We >> need to generate CPU configurations that actually work, i.e., we need >> to ask QEMU what CPU it can provide on current host rather than >> requesting a bunch of features on top of a CPU model which does not >> always match the host CPU. >> >> QEMU already provides or will soon provide everything we need to meet >> these requirements: >> - we can cover every configurable part of a CPU in our cpu_map.xml and >> instead of asking QEMU for a specific CPU model we can use "-cpu >> custom" with a fully specified CPU >> - we can use the additional data about CPU models to choose the right >> one for a host CPU >> - when starting a domain we can check whether QEMU filtered out any of >> the features we asked for and refuse to start the domain >> - we can ask QEMU what would "-cpu host" look like and use that for >> host-model and minimum match CPUs (it won't work for TCG mode, though, >> but we can keep using the current CPUID detection code for TCG) > > In TCG mode of course, 'host-model' and 'host-passthrough' are > effectively identical, and don't actually need the host to support > all the featues, since TCG is fully emulated. Which means that you > can migrated TCG guests to anyhost with any model :-) I wonder if > we are probably accidentally restricting that today, becuase we > assume KVM needs host support. > >> Once we start maintaining CPU models with all the details, we will >> likely meet the same issues QEMU folks meet, i.e., we will need to fix >> bugs in existing CPU models. And it's not just about adding removing CPU >> features but also fixing other parameters, such as wrong level, etc. >> It's clear every change will require a new CPU model to be defined. But >> I think we should do it in a way that applications or users should not >> need (if they don't want to) to care about it. I'm thinking about doing >> something similar to machine types. Each CPU model could be defined in >> several versions and a CPU specs without a version would be an alias to >> the latest version. > > Agreed, I think that versioning CPU models, independantly of machine > types makes sense. It is probably a little more complex - in most cases > we'd increase the version, but in some cases I think we'd end up wanting > to define new named models. For example, with the recent TSX scenario we > had, using versions would not have been appropriate, because Intel in > fact ship 2 variants of the silicon. So even with with versioning, we > would still have wanted to introduce the noTSX variants of the models. > >> The problem is, we need to maintain backward compatibility and we should >> avoid breaking existing domains (shouldn't we?) which just work even >> though their guest CPUs do not exactly match the domain XML definitions. > > Yep breaking existing domains isn't too pleasant! > >> So either we need to define all existing CPU models in all their >> variants used for various machine types and have a mapping between >> (model without a version, machine type) to a specific version of the >> model (which may be quite hard) or we need to be able to distinguish >> between an existing domain and a new domain with no CPU model version. >> While host-model and host-passthrough CPU modes are easy because they >> are designed to change everytime a domain starts (which means we don't >> need to be able to distinguish between existing and new domains), custom >> CPU mode are tricky. Currently, the only at least a bit reasonable thing >> which came to my mind is to have a new CPU mode, but it still seems >> awkward so please share your ideas if you have any. > > Introducing a new CPU mode feels pretty unpleasant to me. > > Although it will certainly be tedious work, getting details of all the > CPU variants for historical machine types should be doable I think. > >> BTW, I don't think we should try to expose every part of the CPU model >> definitions in domain XML, they should remain hidden behind the CPU >> model name. It would be hard to explain what each of the extra >> parameters mean, each model would have to include them anyway since we >> can't expect users to provide all the details correctly, and once >> visible in domain XML it could encourage users to play with the values. > > Yeah, I don't think we need expose all the raw details. If people really > badly want to be able to customize that, then we should instead look at > how we could better enable the cpu_map.xml file to be admin extensible.

Hi, currently Michael Mueller (IBM) is working on an extension of QEMU to support CPU models for s390x platform. During the discussion on the QEMU mailing list the implementation was done in a more common way to provide support for all platforms. According to that new implementation I have implemented a first version for libvirt to retrieve the CPU model(s) supported by QEMU on s390x. Due to the fact that the discussion is ongoing my prototype is not ready to be tested yet. A short overview about the current prototype I have implemented (QEMU cpu model support patches from Michael Mueller required): 1. During start of libvirt daemon QEMU monitor is used to retrieve the CPU models (i.e. just model names, QEMU handles all other setting like features, etc.) QEMU is supporting. 2. The supported CPU models are stored in libvirt's QEMU capabilities (and stored in the capabilities cache file). 3. Each call of virConnectGetCPUModelNames() (i.e. qemuConnectGetCPUModelNames()) is retrieving the information from QEMU capabilities (cached or not) on s390x platform. All other platforms remain on the currently implemented way to parse the cpu_map.xml. Depending on that implementation all requests to get CPU models (e.g. for CPU model comparison, CPU model listing) will lead to a more appropriate result (e.g. if a QEMU binary is exchanged by a QEMU binary built manually). > > Regards, > Daniel >