[libvirt] [RFC] qemu: Redesigning guest CPU configuration
Hi all (and sorry for the long email), The current way QEMU driver handles guest CPU configuration is not ideal. We detect host CPU capabilities only by querying the CPU and we don't check with QEMU what features it supports. We don't check QEMU's definitions of CPU models, which may be different from libvirt's definitions. All this results in several issues: - guest CPU may change druing migration, save/restore - libvirt may ask for a CPU which QEMU cannot provide; the guest will see a slightly different CPU but libvirt client won't know about it - libvirt may come up with a CPU that doesn't make sense and which won't work for a guest (the guest may even crash) Although usually everything just works, it is very fragile. Since we want to fix all these issues, we need to: - guarantee stable guest ABI (a single domain XML should always results in the same guest ABI). Once a domain is started, its CPU definition should never change (unless someone changes the XML, of course, similar to, e.g. PCI addresses). However, there are a few exceptions: - host-passthrough CPU mode will always result in "-cpu host" - host-model CPU mode should recompute the CPU model on every start, but the CPU must not change during migration - always make sure QEMU provides the CPU we asked for. Starting a domain should fail in case QEMU cannot provide exactly the CPU we asked for. - provide usable host-model mode and custom mode with minimum match. We need to generate CPU configurations that actually work, i.e., we need to ask QEMU what CPU it can provide on current host rather than requesting a bunch of features on top of a CPU model which does not always match the host CPU. QEMU already provides or will soon provide everything we need to meet these requirements: - we can cover every configurable part of a CPU in our cpu_map.xml and instead of asking QEMU for a specific CPU model we can use "-cpu custom" with a fully specified CPU - we can use the additional data about CPU models to choose the right one for a host CPU - when starting a domain we can check whether QEMU filtered out any of the features we asked for and refuse to start the domain - we can ask QEMU what would "-cpu host" look like and use that for host-model and minimum match CPUs (it won't work for TCG mode, though, but we can keep using the current CPUID detection code for TCG) Once we start maintaining CPU models with all the details, we will likely meet the same issues QEMU folks meet, i.e., we will need to fix bugs in existing CPU models. And it's not just about adding removing CPU features but also fixing other parameters, such as wrong level, etc. It's clear every change will require a new CPU model to be defined. But I think we should do it in a way that applications or users should not need (if they don't want to) to care about it. I'm thinking about doing something similar to machine types. Each CPU model could be defined in several versions and a CPU specs without a version would be an alias to the latest version. The problem is, we need to maintain backward compatibility and we should avoid breaking existing domains (shouldn't we?) which just work even though their guest CPUs do not exactly match the domain XML definitions. So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any. BTW, I don't think we should try to expose every part of the CPU model definitions in domain XML, they should remain hidden behind the CPU model name. It would be hard to explain what each of the extra parameters mean, each model would have to include them anyway since we can't expect users to provide all the details correctly, and once visible in domain XML it could encourage users to play with the values. I'm looking forward to any comments or ideas. Jirka
On Wed, Jun 17, 2015 at 05:37:42PM +0200, Jiri Denemark wrote:
Hi all (and sorry for the long email),
The current way QEMU driver handles guest CPU configuration is not ideal. We detect host CPU capabilities only by querying the CPU and we don't check with QEMU what features it supports. We don't check QEMU's definitions of CPU models, which may be different from libvirt's definitions. All this results in several issues:
- guest CPU may change druing migration, save/restore - libvirt may ask for a CPU which QEMU cannot provide; the guest will see a slightly different CPU but libvirt client won't know about it - libvirt may come up with a CPU that doesn't make sense and which won't work for a guest (the guest may even crash)
Although usually everything just works, it is very fragile.
A third issue is that if there is no <cpu> in the guest config, we just delegate CPU choice to QEMU and then ignore any CPU checks when migrating. If libvirt owns the full CPU config, we'd probably want to also decide the default ourselves, so that we will always be able todo migrate CPU checks.
Since we want to fix all these issues, we need to: - guarantee stable guest ABI (a single domain XML should always results in the same guest ABI). Once a domain is started, its CPU definition should never change (unless someone changes the XML, of course, similar to, e.g. PCI addresses). However, there are a few exceptions: - host-passthrough CPU mode will always result in "-cpu host" - host-model CPU mode should recompute the CPU model on every start, but the CPU must not change during migration - always make sure QEMU provides the CPU we asked for. Starting a domain should fail in case QEMU cannot provide exactly the CPU we asked for. - provide usable host-model mode and custom mode with minimum match. We need to generate CPU configurations that actually work, i.e., we need to ask QEMU what CPU it can provide on current host rather than requesting a bunch of features on top of a CPU model which does not always match the host CPU.
QEMU already provides or will soon provide everything we need to meet these requirements: - we can cover every configurable part of a CPU in our cpu_map.xml and instead of asking QEMU for a specific CPU model we can use "-cpu custom" with a fully specified CPU - we can use the additional data about CPU models to choose the right one for a host CPU - when starting a domain we can check whether QEMU filtered out any of the features we asked for and refuse to start the domain - we can ask QEMU what would "-cpu host" look like and use that for host-model and minimum match CPUs (it won't work for TCG mode, though, but we can keep using the current CPUID detection code for TCG)
In TCG mode of course, 'host-model' and 'host-passthrough' are effectively identical, and don't actually need the host to support all the featues, since TCG is fully emulated. Which means that you can migrated TCG guests to anyhost with any model :-) I wonder if we are probably accidentally restricting that today, becuase we assume KVM needs host support.
Once we start maintaining CPU models with all the details, we will likely meet the same issues QEMU folks meet, i.e., we will need to fix bugs in existing CPU models. And it's not just about adding removing CPU features but also fixing other parameters, such as wrong level, etc. It's clear every change will require a new CPU model to be defined. But I think we should do it in a way that applications or users should not need (if they don't want to) to care about it. I'm thinking about doing something similar to machine types. Each CPU model could be defined in several versions and a CPU specs without a version would be an alias to the latest version.
Agreed, I think that versioning CPU models, independantly of machine types makes sense. It is probably a little more complex - in most cases we'd increase the version, but in some cases I think we'd end up wanting to define new named models. For example, with the recent TSX scenario we had, using versions would not have been appropriate, because Intel in fact ship 2 variants of the silicon. So even with with versioning, we would still have wanted to introduce the noTSX variants of the models.
The problem is, we need to maintain backward compatibility and we should avoid breaking existing domains (shouldn't we?) which just work even though their guest CPUs do not exactly match the domain XML definitions.
Yep breaking existing domains isn't too pleasant!
So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
Introducing a new CPU mode feels pretty unpleasant to me. Although it will certainly be tedious work, getting details of all the CPU variants for historical machine types should be doable I think.
BTW, I don't think we should try to expose every part of the CPU model definitions in domain XML, they should remain hidden behind the CPU model name. It would be hard to explain what each of the extra parameters mean, each model would have to include them anyway since we can't expect users to provide all the details correctly, and once visible in domain XML it could encourage users to play with the values.
Yeah, I don't think we need expose all the raw details. If people really badly want to be able to customize that, then we should instead look at how we could better enable the cpu_map.xml file to be admin extensible. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 18.06.2015 15:41, Daniel P. Berrange wrote:
On Wed, Jun 17, 2015 at 05:37:42PM +0200, Jiri Denemark wrote:
Hi all (and sorry for the long email),
The current way QEMU driver handles guest CPU configuration is not ideal. We detect host CPU capabilities only by querying the CPU and we don't check with QEMU what features it supports. We don't check QEMU's definitions of CPU models, which may be different from libvirt's definitions. All this results in several issues:
- guest CPU may change druing migration, save/restore - libvirt may ask for a CPU which QEMU cannot provide; the guest will see a slightly different CPU but libvirt client won't know about it - libvirt may come up with a CPU that doesn't make sense and which won't work for a guest (the guest may even crash)
Although usually everything just works, it is very fragile.
A third issue is that if there is no <cpu> in the guest config, we just delegate CPU choice to QEMU and then ignore any CPU checks when migrating. If libvirt owns the full CPU config, we'd probably want to also decide the default ourselves, so that we will always be able todo migrate CPU checks.
Since we want to fix all these issues, we need to: - guarantee stable guest ABI (a single domain XML should always results in the same guest ABI). Once a domain is started, its CPU definition should never change (unless someone changes the XML, of course, similar to, e.g. PCI addresses). However, there are a few exceptions: - host-passthrough CPU mode will always result in "-cpu host" - host-model CPU mode should recompute the CPU model on every start, but the CPU must not change during migration - always make sure QEMU provides the CPU we asked for. Starting a domain should fail in case QEMU cannot provide exactly the CPU we asked for. - provide usable host-model mode and custom mode with minimum match. We need to generate CPU configurations that actually work, i.e., we need to ask QEMU what CPU it can provide on current host rather than requesting a bunch of features on top of a CPU model which does not always match the host CPU.
QEMU already provides or will soon provide everything we need to meet these requirements: - we can cover every configurable part of a CPU in our cpu_map.xml and instead of asking QEMU for a specific CPU model we can use "-cpu custom" with a fully specified CPU - we can use the additional data about CPU models to choose the right one for a host CPU - when starting a domain we can check whether QEMU filtered out any of the features we asked for and refuse to start the domain - we can ask QEMU what would "-cpu host" look like and use that for host-model and minimum match CPUs (it won't work for TCG mode, though, but we can keep using the current CPUID detection code for TCG)
In TCG mode of course, 'host-model' and 'host-passthrough' are effectively identical, and don't actually need the host to support all the featues, since TCG is fully emulated. Which means that you can migrated TCG guests to anyhost with any model :-) I wonder if we are probably accidentally restricting that today, becuase we assume KVM needs host support.
Once we start maintaining CPU models with all the details, we will likely meet the same issues QEMU folks meet, i.e., we will need to fix bugs in existing CPU models. And it's not just about adding removing CPU features but also fixing other parameters, such as wrong level, etc. It's clear every change will require a new CPU model to be defined. But I think we should do it in a way that applications or users should not need (if they don't want to) to care about it. I'm thinking about doing something similar to machine types. Each CPU model could be defined in several versions and a CPU specs without a version would be an alias to the latest version.
Agreed, I think that versioning CPU models, independantly of machine types makes sense. It is probably a little more complex - in most cases we'd increase the version, but in some cases I think we'd end up wanting to define new named models. For example, with the recent TSX scenario we had, using versions would not have been appropriate, because Intel in fact ship 2 variants of the silicon. So even with with versioning, we would still have wanted to introduce the noTSX variants of the models.
The problem is, we need to maintain backward compatibility and we should avoid breaking existing domains (shouldn't we?) which just work even though their guest CPUs do not exactly match the domain XML definitions.
Yep breaking existing domains isn't too pleasant!
So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
Introducing a new CPU mode feels pretty unpleasant to me.
Although it will certainly be tedious work, getting details of all the CPU variants for historical machine types should be doable I think.
BTW, I don't think we should try to expose every part of the CPU model definitions in domain XML, they should remain hidden behind the CPU model name. It would be hard to explain what each of the extra parameters mean, each model would have to include them anyway since we can't expect users to provide all the details correctly, and once visible in domain XML it could encourage users to play with the values.
Yeah, I don't think we need expose all the raw details. If people really badly want to be able to customize that, then we should instead look at how we could better enable the cpu_map.xml file to be admin extensible.
Hi, currently Michael Mueller (IBM) is working on an extension of QEMU to support CPU models for s390x platform. During the discussion on the QEMU mailing list the implementation was done in a more common way to provide support for all platforms. According to that new implementation I have implemented a first version for libvirt to retrieve the CPU model(s) supported by QEMU on s390x. Due to the fact that the discussion is ongoing my prototype is not ready to be tested yet. A short overview about the current prototype I have implemented (QEMU cpu model support patches from Michael Mueller required): 1. During start of libvirt daemon QEMU monitor is used to retrieve the CPU models (i.e. just model names, QEMU handles all other setting like features, etc.) QEMU is supporting. 2. The supported CPU models are stored in libvirt's QEMU capabilities (and stored in the capabilities cache file). 3. Each call of virConnectGetCPUModelNames() (i.e. qemuConnectGetCPUModelNames()) is retrieving the information from QEMU capabilities (cached or not) on s390x platform. All other platforms remain on the currently implemented way to parse the cpu_map.xml. Depending on that implementation all requests to get CPU models (e.g. for CPU model comparison, CPU model listing) will lead to a more appropriate result (e.g. if a QEMU binary is exchanged by a QEMU binary built manually).
Regards, Daniel
-- Mit freundlichen Grüßen / Kind regards Daniel Hansel IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On 19.06.2015 14:27, Daniel Hansel wrote:
On 18.06.2015 15:41, Daniel P. Berrange wrote:
On Wed, Jun 17, 2015 at 05:37:42PM +0200, Jiri Denemark wrote:
Hi all (and sorry for the long email),
The current way QEMU driver handles guest CPU configuration is not ideal. We detect host CPU capabilities only by querying the CPU and we don't check with QEMU what features it supports. We don't check QEMU's definitions of CPU models, which may be different from libvirt's definitions. All this results in several issues:
- guest CPU may change druing migration, save/restore - libvirt may ask for a CPU which QEMU cannot provide; the guest will see a slightly different CPU but libvirt client won't know about it - libvirt may come up with a CPU that doesn't make sense and which won't work for a guest (the guest may even crash)
Although usually everything just works, it is very fragile.
A third issue is that if there is no <cpu> in the guest config, we just delegate CPU choice to QEMU and then ignore any CPU checks when migrating. If libvirt owns the full CPU config, we'd probably want to also decide the default ourselves, so that we will always be able todo migrate CPU checks.
Since we want to fix all these issues, we need to: - guarantee stable guest ABI (a single domain XML should always results in the same guest ABI). Once a domain is started, its CPU definition should never change (unless someone changes the XML, of course, similar to, e.g. PCI addresses). However, there are a few exceptions: - host-passthrough CPU mode will always result in "-cpu host" - host-model CPU mode should recompute the CPU model on every start, but the CPU must not change during migration - always make sure QEMU provides the CPU we asked for. Starting a domain should fail in case QEMU cannot provide exactly the CPU we asked for. - provide usable host-model mode and custom mode with minimum match. We need to generate CPU configurations that actually work, i.e., we need to ask QEMU what CPU it can provide on current host rather than requesting a bunch of features on top of a CPU model which does not always match the host CPU.
QEMU already provides or will soon provide everything we need to meet these requirements: - we can cover every configurable part of a CPU in our cpu_map.xml and instead of asking QEMU for a specific CPU model we can use "-cpu custom" with a fully specified CPU - we can use the additional data about CPU models to choose the right one for a host CPU - when starting a domain we can check whether QEMU filtered out any of the features we asked for and refuse to start the domain - we can ask QEMU what would "-cpu host" look like and use that for host-model and minimum match CPUs (it won't work for TCG mode, though, but we can keep using the current CPUID detection code for TCG)
In TCG mode of course, 'host-model' and 'host-passthrough' are effectively identical, and don't actually need the host to support all the featues, since TCG is fully emulated. Which means that you can migrated TCG guests to anyhost with any model :-) I wonder if we are probably accidentally restricting that today, becuase we assume KVM needs host support.
Once we start maintaining CPU models with all the details, we will likely meet the same issues QEMU folks meet, i.e., we will need to fix bugs in existing CPU models. And it's not just about adding removing CPU features but also fixing other parameters, such as wrong level, etc. It's clear every change will require a new CPU model to be defined. But I think we should do it in a way that applications or users should not need (if they don't want to) to care about it. I'm thinking about doing something similar to machine types. Each CPU model could be defined in several versions and a CPU specs without a version would be an alias to the latest version.
Agreed, I think that versioning CPU models, independantly of machine types makes sense. It is probably a little more complex - in most cases we'd increase the version, but in some cases I think we'd end up wanting to define new named models. For example, with the recent TSX scenario we had, using versions would not have been appropriate, because Intel in fact ship 2 variants of the silicon. So even with with versioning, we would still have wanted to introduce the noTSX variants of the models.
The problem is, we need to maintain backward compatibility and we should avoid breaking existing domains (shouldn't we?) which just work even though their guest CPUs do not exactly match the domain XML definitions.
Yep breaking existing domains isn't too pleasant!
So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
Introducing a new CPU mode feels pretty unpleasant to me.
Although it will certainly be tedious work, getting details of all the CPU variants for historical machine types should be doable I think.
BTW, I don't think we should try to expose every part of the CPU model definitions in domain XML, they should remain hidden behind the CPU model name. It would be hard to explain what each of the extra parameters mean, each model would have to include them anyway since we can't expect users to provide all the details correctly, and once visible in domain XML it could encourage users to play with the values.
Yeah, I don't think we need expose all the raw details. If people really badly want to be able to customize that, then we should instead look at how we could better enable the cpu_map.xml file to be admin extensible.
Hi Daniel and Jirka, just as a ping if you have missed my comment...
Hi,
currently Michael Mueller (IBM) is working on an extension of QEMU to support CPU models for s390x platform. During the discussion on the QEMU mailing list the implementation was done in a more common way to provide support for all platforms.
According to that new implementation I have implemented a first version for libvirt to retrieve the CPU model(s) supported by QEMU on s390x. Due to the fact that the discussion is ongoing my prototype is not ready to be tested yet.
A short overview about the current prototype I have implemented (QEMU cpu model support patches from Michael Mueller required):
1. During start of libvirt daemon QEMU monitor is used to retrieve the CPU models (i.e. just model names, QEMU handles all other setting like features, etc.) QEMU is supporting. 2. The supported CPU models are stored in libvirt's QEMU capabilities (and stored in the capabilities cache file). 3. Each call of virConnectGetCPUModelNames() (i.e. qemuConnectGetCPUModelNames()) is retrieving the information from QEMU capabilities (cached or not) on s390x platform. All other platforms remain on the currently implemented way to parse the cpu_map.xml.
Depending on that implementation all requests to get CPU models (e.g. for CPU model comparison, CPU model listing) will lead to a more appropriate result (e.g. if a QEMU binary is exchanged by a QEMU binary built manually).
Regards, Daniel
-- Mit freundlichen Grüßen / Kind regards Daniel Hansel IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Fri, Jun 19, 2015 at 02:27:27PM +0200, Daniel Hansel wrote:
currently Michael Mueller (IBM) is working on an extension of QEMU to support CPU models for s390x platform. During the discussion on the QEMU mailing list the implementation was done in a more common way to provide support for all platforms.
According to that new implementation I have implemented a first version for libvirt to retrieve the CPU model(s) supported by QEMU on s390x. Due to the fact that the discussion is ongoing my prototype is not ready to be tested yet.
A short overview about the current prototype I have implemented (QEMU cpu model support patches from Michael Mueller required):
1. During start of libvirt daemon QEMU monitor is used to retrieve the CPU models (i.e. just model names, QEMU handles all other setting like features, etc.) QEMU is supporting. 2. The supported CPU models are stored in libvirt's QEMU capabilities (and stored in the capabilities cache file). 3. Each call of virConnectGetCPUModelNames() (i.e. qemuConnectGetCPUModelNames()) is retrieving the information from QEMU capabilities (cached or not) on s390x platform. All other platforms remain on the currently implemented way to parse the cpu_map.xml.
Depending on that implementation all requests to get CPU models (e.g. for CPU model comparison, CPU model listing) will lead to a more appropriate result (e.g. if a QEMU binary is exchanged by a QEMU binary built manually).
I can't really speak for Jiri and I haven't look at QEMU s390 in any detail, but as a general point, we'll want to try to use the same approach for dealing with CPUs across all architectures. What Jiri describes has been written from POV of x86, but we need to make sure it works on s390 and other arches just as well. We especially want to avoid having two completely separate codepaths for this in libvirt dependant on arch. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 23.06.2015 11:28, Daniel P. Berrange wrote:
On Fri, Jun 19, 2015 at 02:27:27PM +0200, Daniel Hansel wrote:
currently Michael Mueller (IBM) is working on an extension of QEMU to support CPU models for s390x platform. During the discussion on the QEMU mailing list the implementation was done in a more common way to provide support for all platforms.
According to that new implementation I have implemented a first version for libvirt to retrieve the CPU model(s) supported by QEMU on s390x. Due to the fact that the discussion is ongoing my prototype is not ready to be tested yet.
A short overview about the current prototype I have implemented (QEMU cpu model support patches from Michael Mueller required):
1. During start of libvirt daemon QEMU monitor is used to retrieve the CPU models (i.e. just model names, QEMU handles all other setting like features, etc.) QEMU is supporting. 2. The supported CPU models are stored in libvirt's QEMU capabilities (and stored in the capabilities cache file). 3. Each call of virConnectGetCPUModelNames() (i.e. qemuConnectGetCPUModelNames()) is retrieving the information from QEMU capabilities (cached or not) on s390x platform. All other platforms remain on the currently implemented way to parse the cpu_map.xml.
Depending on that implementation all requests to get CPU models (e.g. for CPU model comparison, CPU model listing) will lead to a more appropriate result (e.g. if a QEMU binary is exchanged by a QEMU binary built manually).
I can't really speak for Jiri and I haven't look at QEMU s390 in any detail, but as a general point, we'll want to try to use the same approach for dealing with CPUs across all architectures. What Jiri describes has been written from POV of x86, but we need to make sure it works on s390 and other arches just as well. We especially want to avoid having two completely separate codepaths for this in libvirt dependant on arch.
The changes in QEMU are made platform independent. Each platform can use/retrieve the information about supported CPU models in the same way. This was the primary requirement for the changes in QEMU Michael Mueller has made. To ensure libvirt is doing its work in the same way it did in the past I have added the new QEMU behavior handling in a separate path (i.e. at the moment for s390 only). But if we decide to use the new information handling of QEMU for all platforms it could be changed easily.
Regards, Daniel
-- Mit freundlichen Grüßen / Kind regards Daniel Hansel IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Fri, Jun 19, 2015 at 14:27:27 +0200, Daniel Hansel wrote: ...
1. During start of libvirt daemon QEMU monitor is used to retrieve the CPU models (i.e. just model names, QEMU handles all other setting like features, etc.) QEMU is supporting.
2. The supported CPU models are stored in libvirt's QEMU capabilities (and stored in the capabilities cache file).
These are fine.
3. Each call of virConnectGetCPUModelNames() (i.e. qemuConnectGetCPUModelNames()) is retrieving the information from QEMU capabilities (cached or not) on s390x platform. All other platforms remain on the currently implemented way to parse the cpu_map.xml.
But this is not. virConnectGetCPUModelNames() was not really designed to provide a list of CPUs supported by a specific emulator binary with a specific machine type. Thus, we should keep virConnectGetCPUModelNames always return a list of CPU models known to libvirt. We should use emulator capabilities XML returned by virConnectGetDomainCapabilities to advertise CPU models supported by QEMU for each combination of (binary, arch, machine, accel) including additional stuff such as whether the CPU is the default one, is migratable, is runnable on current host etc. And we should do this on all achitectures, not just s390x. That is, we need to come up with a good XML design which would fit all architectures.
Depending on that implementation all requests to get CPU models (e.g. for CPU model comparison, CPU model listing) will lead to a more appropriate result (e.g. if a QEMU binary is exchanged by a QEMU binary built manually).
We may need to provide additional CPU related APIs which would take (binary, arch, machine, accel) into account. But if we are good at reporting relevant data in the emulator capabilities XML, the new APIs may be unnecessary. For example, an app/user can just query the XML for all runnable CPU models instead of comparing every CPU model known to libvirt with a current host. Jirka
On Thu, Jun 18, 2015 at 14:41:17 +0100, Daniel P. Berrange wrote:
On Wed, Jun 17, 2015 at 05:37:42PM +0200, Jiri Denemark wrote:
Hi all (and sorry for the long email),
The current way QEMU driver handles guest CPU configuration is not ideal. We detect host CPU capabilities only by querying the CPU and we don't check with QEMU what features it supports. We don't check QEMU's definitions of CPU models, which may be different from libvirt's definitions. All this results in several issues:
- guest CPU may change druing migration, save/restore - libvirt may ask for a CPU which QEMU cannot provide; the guest will see a slightly different CPU but libvirt client won't know about it - libvirt may come up with a CPU that doesn't make sense and which won't work for a guest (the guest may even crash)
Although usually everything just works, it is very fragile.
A third issue is that if there is no <cpu> in the guest config, we just delegate CPU choice to QEMU and then ignore any CPU checks when migrating. If libvirt owns the full CPU config, we'd probably want to also decide the default ourselves, so that we will always be able todo migrate CPU checks.
This is not going to be that easy, I'm afraid. The default CPU model used by QEMU is filtered to only contain features supported by a host CPU (I'm talking about KVM, of course), so the default may vary even though it's the same model. Which means adding an explicit default which exactly matches the implicit one is not very easy. I'll see what we can do about this.
Since we want to fix all these issues, we need to: - guarantee stable guest ABI (a single domain XML should always results in the same guest ABI). Once a domain is started, its CPU definition should never change (unless someone changes the XML, of course, similar to, e.g. PCI addresses). However, there are a few exceptions: - host-passthrough CPU mode will always result in "-cpu host" - host-model CPU mode should recompute the CPU model on every start, but the CPU must not change during migration - always make sure QEMU provides the CPU we asked for. Starting a domain should fail in case QEMU cannot provide exactly the CPU we asked for. - provide usable host-model mode and custom mode with minimum match. We need to generate CPU configurations that actually work, i.e., we need to ask QEMU what CPU it can provide on current host rather than requesting a bunch of features on top of a CPU model which does not always match the host CPU.
QEMU already provides or will soon provide everything we need to meet these requirements: - we can cover every configurable part of a CPU in our cpu_map.xml and instead of asking QEMU for a specific CPU model we can use "-cpu custom" with a fully specified CPU - we can use the additional data about CPU models to choose the right one for a host CPU - when starting a domain we can check whether QEMU filtered out any of the features we asked for and refuse to start the domain - we can ask QEMU what would "-cpu host" look like and use that for host-model and minimum match CPUs (it won't work for TCG mode, though, but we can keep using the current CPUID detection code for TCG)
In TCG mode of course, 'host-model' and 'host-passthrough' are effectively identical, and don't actually need the host to support all the featues, since TCG is fully emulated. Which means that you can migrated TCG guests to anyhost with any model :-) I wonder if we are probably accidentally restricting that today, becuase we assume KVM needs host support.
Yeah, I think we treat TCG similarly to KVM. At least we did that in the past. I'm planning to look at it and fix it in my series.
Once we start maintaining CPU models with all the details, we will likely meet the same issues QEMU folks meet, i.e., we will need to fix bugs in existing CPU models. And it's not just about adding removing CPU features but also fixing other parameters, such as wrong level, etc. It's clear every change will require a new CPU model to be defined. But I think we should do it in a way that applications or users should not need (if they don't want to) to care about it. I'm thinking about doing something similar to machine types. Each CPU model could be defined in several versions and a CPU specs without a version would be an alias to the latest version.
Agreed, I think that versioning CPU models, independantly of machine types makes sense. It is probably a little more complex - in most cases we'd increase the version, but in some cases I think we'd end up wanting to define new named models. For example, with the recent TSX scenario we had, using versions would not have been appropriate, because Intel in fact ship 2 variants of the silicon. So even with with versioning, we would still have wanted to introduce the noTSX variants of the models.
The problem is, we need to maintain backward compatibility and we should avoid breaking existing domains (shouldn't we?) which just work even though their guest CPUs do not exactly match the domain XML definitions.
Yep breaking existing domains isn't too pleasant!
So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
Introducing a new CPU mode feels pretty unpleasant to me.
Although it will certainly be tedious work, getting details of all the CPU variants for historical machine types should be doable I think.
Yeah, I also prefer this variant but I was kind of hoping someone would come up with a bright idea which would safe me from all the work :-P
BTW, I don't think we should try to expose every part of the CPU model definitions in domain XML, they should remain hidden behind the CPU model name. It would be hard to explain what each of the extra parameters mean, each model would have to include them anyway since we can't expect users to provide all the details correctly, and once visible in domain XML it could encourage users to play with the values.
Yeah, I don't think we need expose all the raw details. If people really badly want to be able to customize that, then we should instead look at how we could better enable the cpu_map.xml file to be admin extensible.
Great, looks like we're in agreement then. Jirka
On Fri, Jun 19, 2015 at 04:36:34PM +0200, Jiri Denemark wrote:
On Thu, Jun 18, 2015 at 14:41:17 +0100, Daniel P. Berrange wrote:
So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
Introducing a new CPU mode feels pretty unpleasant to me.
Although it will certainly be tedious work, getting details of all the CPU variants for historical machine types should be doable I think.
Yeah, I also prefer this variant but I was kind of hoping someone would come up with a bright idea which would safe me from all the work :-P
Allow me to introduce you to perl and regexes :-P Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Jun 19, 2015 at 15:43:02 +0100, Daniel P. Berrange wrote:
On Fri, Jun 19, 2015 at 04:36:34PM +0200, Jiri Denemark wrote:
On Thu, Jun 18, 2015 at 14:41:17 +0100, Daniel P. Berrange wrote:
So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
Introducing a new CPU mode feels pretty unpleasant to me.
Although it will certainly be tedious work, getting details of all the CPU variants for historical machine types should be doable I think.
Yeah, I also prefer this variant but I was kind of hoping someone would come up with a bright idea which would safe me from all the work :-P
Allow me to introduce you to perl and regexes :-P
Haha, I'm afraid it's not as simple since the machine type specific tweaks to CPU models are done in the code, you can't just look at some data structures to get all the variants. Actually, it has changed and it's defined in macros now, but there is another way which does not need a lot of work :-) Using a slightly modified (to produce a bit more condensed output) x86-cpu-model-dump script from Eduardo and some shell scripting around it I fetched all 700+ combinations of CPU models and machine types and differences between subsequent variants of the same CPU model. So this part was not that bad. However, knowing all the details about a guest CPU used by QEMU for a given CPU model on a specific machine type is not enough to enforce ABI stability. Without using -cpu Model,enforce (or an equivalent of checking filtered-features via QMP) QEMU may silently filter features it cannot provide on the current host. Even in case of TCG some features are not supported, e.g., -cpu SandyBridge will always fail to start in enforcing mode. Even doing something ugly and using the enforce mode only for new machine types is not going to work because after a QEMU upgrade new libvirt would be incompatible with older libvirt. That said, I don't really see a way to do all this automatically without an explicit switch in a domain XML. Be it a new CPU mode or an attribute which would request enforcing ABI stability. Jirka
On Mon, Jun 22, 2015 at 05:58:46PM +0200, Jiri Denemark wrote:
However, knowing all the details about a guest CPU used by QEMU for a given CPU model on a specific machine type is not enough to enforce ABI stability. Without using -cpu Model,enforce (or an equivalent of checking filtered-features via QMP) QEMU may silently filter features it cannot provide on the current host. Even in case of TCG some features are not supported, e.g., -cpu SandyBridge will always fail to start in enforcing mode. Even doing something ugly and using the enforce mode only for new machine types is not going to work because after a QEMU upgrade new libvirt would be incompatible with older libvirt.
I'm not sure I follow the scenario you're concerned with. Lets, say we have guest XML <cpu><model>SandyBridge</model></cpu> and so we're using the new "custom" -cpu arg that QEMU supports. Are you saying that we won't be able to live migrate from the "-cpu custom" with new QEMU, to "-cpu SandyBridge" with old QEMU, even if the CPU seen by the guest is identical ?
That said, I don't really see a way to do all this automatically without an explicit switch in a domain XML. Be it a new CPU mode or an attribute which would request enforcing ABI stability.
I don't like the idea of adding more to the mode=custom|host-model|passthroug options, but perhaps we could signify this in a differnet way. For example, what we're really doing here is switching between use of libvirt and use of QEMU for CPU emulation. In similar cases, for other device types we use the <driver> element to identify the backend impl. So perhaps we could do a <cpu> ... <driver name="libvirt|qemu"/> </cpu> To distinguish between use of libvirt and use of QEMU for the CPU model/feature handling ? Ideally if not specified, then we'd magically choose the "best" approach given the QEMU we have available Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Jun 22, 2015 at 17:09:22 +0100, Daniel P. Berrange wrote:
On Mon, Jun 22, 2015 at 05:58:46PM +0200, Jiri Denemark wrote:
However, knowing all the details about a guest CPU used by QEMU for a given CPU model on a specific machine type is not enough to enforce ABI stability. Without using -cpu Model,enforce (or an equivalent of checking filtered-features via QMP) QEMU may silently filter features it cannot provide on the current host. Even in case of TCG some features are not supported, e.g., -cpu SandyBridge will always fail to start in enforcing mode. Even doing something ugly and using the enforce mode only for new machine types is not going to work because after a QEMU upgrade new libvirt would be incompatible with older libvirt.
I'm not sure I follow the scenario you're concerned with.
Lets, say we have guest XML <cpu><model>SandyBridge</model></cpu> and so we're using the new "custom" -cpu arg that QEMU supports. Are you saying that we won't be able to live migrate from the "-cpu custom" with new QEMU, to "-cpu SandyBridge" with old QEMU, even if the CPU seen by the guest is identical ?
There are two more or less separate issues. The first one is switching from -cpu SandyBridge to -cpu custom. This should be safe and having the mapping between machine types and CPU model version should make both command lines compatible even during migration. The second one is adding enforce, i.e., -cpu SandyBridge,enforce or -cpu custom,enforce (libvirt will likely implement it in a different way but the effect will be the same) to make sure QEMU does not filter any CPU feature. Without "enforce", QEMU may silently drop any feature requested by the SandyBridge CPU model. During migration, QEMU on both sides can filter different features in case the host CPUs, kernel versions or settings, QEMU versions, or BIOS settings differ. So we really need to start making sure QEMU does not filter anything. But we shouldn't do that for existing domains because they could fail to start or migrate because QEMU filtered some features and we didn't care so far.
That said, I don't really see a way to do all this automatically without an explicit switch in a domain XML. Be it a new CPU mode or an attribute which would request enforcing ABI stability.
I don't like the idea of adding more to the mode=custom|host-model|passthroug options, but perhaps we could signify this in a differnet way.
I'm not a big fan of this either.
For example, what we're really doing here is switching between use of libvirt and use of QEMU for CPU emulation. In similar cases, for other device types we use the <driver> element to identify the backend impl. So perhaps we could do a
<cpu> ... <driver name="libvirt|qemu"/> </cpu>
To distinguish between use of libvirt and use of QEMU for the CPU model/feature handling ? Ideally if not specified, then we'd magically choose the "best" approach given the QEMU we have available
This would cover the first issue (-cpu Model vs. -cpu custom), which I think can be done automatically and we shouldn't need any XML modifications for it. I'm more concerned about the second issue (enforcing ABI stability). We could automatically enable enforcing mode for new CPU models, but we need and explicit switch for existing models. I was thinking about an attribute for <cpu> or maybe even batter for <model> which would turn enforcing on. Something like <cpu mode='custom' match='exact'> <model fallback='allow' check='strict|relaxed'>...</model> ... </cpu> Any CPU model which is not currently known to libvirt could easily default to check='strict'. Jirka
On Mon, Jun 22, 2015 at 18:43:56 +0200, Jiri Denemark wrote:
On Mon, Jun 22, 2015 at 17:09:22 +0100, Daniel P. Berrange wrote:
On Mon, Jun 22, 2015 at 05:58:46PM +0200, Jiri Denemark wrote:
However, knowing all the details about a guest CPU used by QEMU for a given CPU model on a specific machine type is not enough to enforce ABI stability. Without using -cpu Model,enforce (or an equivalent of checking filtered-features via QMP) QEMU may silently filter features it cannot provide on the current host. Even in case of TCG some features are not supported, e.g., -cpu SandyBridge will always fail to start in enforcing mode. Even doing something ugly and using the enforce mode only for new machine types is not going to work because after a QEMU upgrade new libvirt would be incompatible with older libvirt.
I'm not sure I follow the scenario you're concerned with.
Lets, say we have guest XML <cpu><model>SandyBridge</model></cpu> and so we're using the new "custom" -cpu arg that QEMU supports. Are you saying that we won't be able to live migrate from the "-cpu custom" with new QEMU, to "-cpu SandyBridge" with old QEMU, even if the CPU seen by the guest is identical ?
There are two more or less separate issues. The first one is switching from -cpu SandyBridge to -cpu custom. This should be safe and having the mapping between machine types and CPU model version should make both command lines compatible even during migration.
The second one is adding enforce, i.e., -cpu SandyBridge,enforce or -cpu custom,enforce (libvirt will likely implement it in a different way but the effect will be the same) to make sure QEMU does not filter any CPU feature. Without "enforce", QEMU may silently drop any feature requested by the SandyBridge CPU model. During migration, QEMU on both sides can filter different features in case the host CPUs, kernel versions or settings, QEMU versions, or BIOS settings differ. So we really need to start making sure QEMU does not filter anything. But we shouldn't do that for existing domains because they could fail to start or migrate because QEMU filtered some features and we didn't care so far.
That said, I don't really see a way to do all this automatically without an explicit switch in a domain XML. Be it a new CPU mode or an attribute which would request enforcing ABI stability.
I don't like the idea of adding more to the mode=custom|host-model|passthroug options, but perhaps we could signify this in a differnet way.
I'm not a big fan of this either.
For example, what we're really doing here is switching between use of libvirt and use of QEMU for CPU emulation. In similar cases, for other device types we use the <driver> element to identify the backend impl. So perhaps we could do a
<cpu> ... <driver name="libvirt|qemu"/> </cpu>
To distinguish between use of libvirt and use of QEMU for the CPU model/feature handling ? Ideally if not specified, then we'd magically choose the "best" approach given the QEMU we have available
This would cover the first issue (-cpu Model vs. -cpu custom), which I think can be done automatically and we shouldn't need any XML modifications for it.
I'm more concerned about the second issue (enforcing ABI stability). We could automatically enable enforcing mode for new CPU models, but we need and explicit switch for existing models. I was thinking about an attribute for <cpu> or maybe even batter for <model> which would turn enforcing on. Something like
<cpu mode='custom' match='exact'> <model fallback='allow' check='strict|relaxed'>...</model> ... </cpu>
Any CPU model which is not currently known to libvirt could easily default to check='strict'.
Also any mode != custom or match == minimal would be strict by default since it's us specifying the exact CPU model. Jirka
On Mon, Jun 22, 2015 at 18:43:56 +0200, Jiri Denemark wrote:
On Mon, Jun 22, 2015 at 17:09:22 +0100, Daniel P. Berrange wrote:
On Mon, Jun 22, 2015 at 05:58:46PM +0200, Jiri Denemark wrote:
However, knowing all the details about a guest CPU used by QEMU for a given CPU model on a specific machine type is not enough to enforce ABI stability. Without using -cpu Model,enforce (or an equivalent of checking filtered-features via QMP) QEMU may silently filter features it cannot provide on the current host. Even in case of TCG some features are not supported, e.g., -cpu SandyBridge will always fail to start in enforcing mode. Even doing something ugly and using the enforce mode only for new machine types is not going to work because after a QEMU upgrade new libvirt would be incompatible with older libvirt.
I'm not sure I follow the scenario you're concerned with.
Lets, say we have guest XML <cpu><model>SandyBridge</model></cpu> and so we're using the new "custom" -cpu arg that QEMU supports. Are you saying that we won't be able to live migrate from the "-cpu custom" with new QEMU, to "-cpu SandyBridge" with old QEMU, even if the CPU seen by the guest is identical ?
There are two more or less separate issues. The first one is switching from -cpu SandyBridge to -cpu custom. This should be safe and having the mapping between machine types and CPU model version should make both command lines compatible even during migration.
The second one is adding enforce, i.e., -cpu SandyBridge,enforce or -cpu custom,enforce (libvirt will likely implement it in a different way but the effect will be the same) to make sure QEMU does not filter any CPU feature. Without "enforce", QEMU may silently drop any feature requested by the SandyBridge CPU model. During migration, QEMU on both sides can filter different features in case the host CPUs, kernel versions or settings, QEMU versions, or BIOS settings differ. So we really need to start making sure QEMU does not filter anything. But we shouldn't do that for existing domains because they could fail to start or migrate because QEMU filtered some features and we didn't care so far.
That said, I don't really see a way to do all this automatically without an explicit switch in a domain XML. Be it a new CPU mode or an attribute which would request enforcing ABI stability.
I don't like the idea of adding more to the mode=custom|host-model|passthroug options, but perhaps we could signify this in a differnet way.
I'm not a big fan of this either.
For example, what we're really doing here is switching between use of libvirt and use of QEMU for CPU emulation. In similar cases, for other device types we use the <driver> element to identify the backend impl. So perhaps we could do a
<cpu> ... <driver name="libvirt|qemu"/> </cpu>
To distinguish between use of libvirt and use of QEMU for the CPU model/feature handling ? Ideally if not specified, then we'd magically choose the "best" approach given the QEMU we have available
This would cover the first issue (-cpu Model vs. -cpu custom), which I think can be done automatically and we shouldn't need any XML modifications for it.
I'm more concerned about the second issue (enforcing ABI stability). We could automatically enable enforcing mode for new CPU models, but we need and explicit switch for existing models. I was thinking about an attribute for <cpu> or maybe even batter for <model> which would turn enforcing on. Something like
<cpu mode='custom' match='exact'> <model fallback='allow' check='strict|relaxed'>...</model> ... </cpu>
Any CPU model which is not currently known to libvirt could easily default to check='strict'.
Actually, we could automatically update domain XML when reconnecting to it on libvirtd restart or when the domain is first started and remove all features filtered by QEMU from it and enforce it doesn't change on migration, save/restore, or next start. So we could turned check from relaxed to strict automatically even for old domains (but not on their first start). Jirka
On Wed, 17 Jun 2015 17:37:42 +0200 Jiri Denemark <jdenemar@redhat.com> wrote:
Hi all (and sorry for the long email),
The current way QEMU driver handles guest CPU configuration is not ideal. We detect host CPU capabilities only by querying the CPU and we don't check with QEMU what features it supports. We don't check QEMU's definitions of CPU models, which may be different from libvirt's definitions. All this results in several issues:
- guest CPU may change druing migration, save/restore - libvirt may ask for a CPU which QEMU cannot provide; the guest will see a slightly different CPU but libvirt client won't know about it - libvirt may come up with a CPU that doesn't make sense and which won't work for a guest (the guest may even crash)
Although usually everything just works, it is very fragile.
Since we want to fix all these issues, we need to: - guarantee stable guest ABI (a single domain XML should always results in the same guest ABI). Once a domain is started, its CPU definition should never change (unless someone changes the XML, of course, similar to, e.g. PCI addresses). However, there are a few exceptions: - host-passthrough CPU mode will always result in "-cpu host" - host-model CPU mode should recompute the CPU model on every start, but the CPU must not change during migration - always make sure QEMU provides the CPU we asked for. Starting a domain should fail in case QEMU cannot provide exactly the CPU we asked for. - provide usable host-model mode and custom mode with minimum match. We need to generate CPU configurations that actually work, i.e., we need to ask QEMU what CPU it can provide on current host rather than requesting a bunch of features on top of a CPU model which does not always match the host CPU.
QEMU already provides or will soon provide everything we need to meet these requirements: - we can cover every configurable part of a CPU in our cpu_map.xml and instead of asking QEMU for a specific CPU model we can use "-cpu custom" with a fully specified CPU - we can use the additional data about CPU models to choose the right one for a host CPU - when starting a domain we can check whether QEMU filtered out any of the features we asked for and refuse to start the domain - we can ask QEMU what would "-cpu host" look like and use that for host-model and minimum match CPUs (it won't work for TCG mode, though, but we can keep using the current CPUID detection code for TCG)
Once we start maintaining CPU models with all the details, we will likely meet the same issues QEMU folks meet, i.e., we will need to fix bugs in existing CPU models. And it's not just about adding removing CPU features but also fixing other parameters, such as wrong level, etc. It's clear every change will require a new CPU model to be defined. But I think we should do it in a way that applications or users should not need (if they don't want to) to care about it. I'm thinking about doing something similar to machine types. Each CPU model could be defined in several versions and a CPU specs without a version would be an alias to the latest version.
The problem is, we need to maintain backward compatibility and we should avoid breaking existing domains (shouldn't we?) which just work even though their guest CPUs do not exactly match the domain XML definitions. So either we need to define all existing CPU models in all their variants used for various machine types and have a mapping between (model without a version, machine type) to a specific version of the model (which may be quite hard) or we need to be able to distinguish between an existing domain and a new domain with no CPU model version. While host-model and host-passthrough CPU modes are easy because they are designed to change everytime a domain starts (which means we don't need to be able to distinguish between existing and new domains), custom CPU mode are tricky. Currently, the only at least a bit reasonable thing which came to my mind is to have a new CPU mode, but it still seems awkward so please share your ideas if you have any.
BTW, I don't think we should try to expose every part of the CPU model definitions in domain XML, they should remain hidden behind the CPU model name. It would be hard to explain what each of the extra parameters mean, each model would have to include them anyway since we can't expect users to provide all the details correctly, and once visible in domain XML it could encourage users to play with the values.
I'm looking forward to any comments or ideas.
Hi Jiri, I basically agree with your analysis on the current situation but are in doubt that the below sketched proposal will overcome the issue. Beside the CPU model and its features also the QEMU level (-machine <version>), the accellerator type (-accel KVM, TCG, ...) also the kernel (KVM) and the host CPU will determine which CPU "capabilities" will be available and what cpu models are runnable. I'm proposing with this patch: "Add optional parameters to QMP command query-cpu-definitions" https://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg03440.html to extend the information returned by query-cpu-definition such that it will allow to cope the stable guest ABI related issues by marking cpu models accordingly to be consumable by libvirt or others. s390 just happens to be the first target that will see an implementation of these additional attributes: "target-s390x: Extend arch specific QMP command query-cpu-definitions" https://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg03449.html Please have short look to the proposed changes of query-cpu-definitions and share your opinion from the libvirt point of view. Thanks, Michael
Jirka
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
participants (4)
-
Daniel Hansel -
Daniel P. Berrange -
Jiri Denemark -
Michael Mueller