[libvirt] Question about the host-model CPU mode
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain). Is this behavior expected? There are some ways to avoid this problem: - CPU model expansion is performed at domain definition time - host-model is related to the machine type of a domain Thank you in advance. Beste Grüße / Kind regards Marc Hartmayer IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Thu, Oct 05, 2017 at 10:46:24 +0200, Marc Hartmayer wrote:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
If I understand it correctly, when QEMU 2.10 is installed, we expand the host-model CPU into a different model than with QEMU 2.9. And the CPU model we got with QEMU 2.10 does not work with s390-virtio-ccw-2.9 machine type even on QEMU 2.10. Is that correct?
Is this behavior expected?
Partially.
There are some ways to avoid this problem: - CPU model expansion is performed at domain definition time
No, host-model is designed to be expanded when a domain starts so that the same definition always results in the best CPU model to be used no matter on which host the domain gest started.
- host-model is related to the machine type of a domain
So if usability of a specific CPU model depends on the selected machine types, we probably need to count with this restriction. But it's going to be a bit complicated because we ask QEMU what the host CPU is and the interface we used would need to be enhanced to give us different results for all supported machine types so that we can select the right one when a domain is started. Jirka
On Thu, Oct 05, 2017 at 02:11 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
On Thu, Oct 05, 2017 at 10:46:24 +0200, Marc Hartmayer wrote:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
If I understand it correctly, when QEMU 2.10 is installed, we expand the host-model CPU into a different model than with QEMU 2.9. And the CPU model we got with QEMU 2.10 does not work with s390-virtio-ccw-2.9 machine type even on QEMU 2.10. Is that correct?
Right.
Is this behavior expected?
Partially.
There are some ways to avoid this problem: - CPU model expansion is performed at domain definition time
No, host-model is designed to be expanded when a domain starts so that the same definition always results in the best CPU model to be used no matter on which host the domain gest started.
- host-model is related to the machine type of a domain
So if usability of a specific CPU model depends on the selected machine types, we probably need to count with this restriction.
Yep.
But it's going to be a bit complicated because we ask QEMU what the host CPU is and the interface we used would need to be enhanced to give us different results for all supported machine types so that we can select the right one when a domain is started.
So how do we deal with this?
Jirka
-- Beste Grüße / Kind regards Marc Hartmayer IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Mon, Oct 09, 2017 at 10:16:48 +0200, Marc Hartmayer wrote:
On Thu, Oct 05, 2017 at 02:11 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
But it's going to be a bit complicated because we ask QEMU what the host CPU is and the interface we used would need to be enhanced to give us different results for all supported machine types so that we can select the right one when a domain is started.
So how do we deal with this?
I think we need to start discussing this on qemu-devel list. If I remember correctly, David Hildenbrand is the original author of the query-cpu-model-expansion QMP command. Please, Cc me so that the thread is more visible to me. Thanks, Jirka
On Thu, Oct 12, 2017 at 02:07 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
On Mon, Oct 09, 2017 at 10:16:48 +0200, Marc Hartmayer wrote:
On Thu, Oct 05, 2017 at 02:11 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
But it's going to be a bit complicated because we ask QEMU what the host CPU is and the interface we used would need to be enhanced to give us different results for all supported machine types so that we can select the right one when a domain is started.
So how do we deal with this?
I think we need to start discussing this on qemu-devel list. If I remember correctly, David Hildenbrand is the original author of the query-cpu-model-expansion QMP command.
Please, Cc me so that the thread is more visible to me.
Thanks,
Jirka
Hi all, we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem: Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain). For determining the actual host-model the QMP command 'query-cpu-model-expansion' is used. This is done once per QEMU binary and the result of it is cached by libvirt. The problem with that is that libvirt queries with the newest machine type of the QEMU binary for the host CPU model. But now we realized that we have to consider the machine type of each domain for the determination of the host CPU model of a domain. We could now either probe the host CPU model for each QEMU binary + machine type combination and for this we've to start a new QEMU process each time. Or we can add a QMP command/parameter that allows us to determine the host CPU model(s) with respect to the passed machine type and/or all supported machine types. The QMP command query-cpu-model-expansion is currently described in qapi-schema.json as follows: <quotation> ## # @query-cpu-model-expansion: # # Expands a given CPU model (or a combination of CPU model + additional options) # to different granularities, allowing tooling to get an understanding what a # specific CPU model looks like in QEMU under a certain configuration. # # This interface can be used to query the "host" CPU model. # # The data returned by this command may be affected by: # # * QEMU version: CPU models may look different depending on the QEMU version. # (Except for CPU models reported as "static" in query-cpu-definitions.) # * machine-type: CPU model may look different depending on the machine-type. # (Except for CPU models reported as "static" in query-cpu-definitions.) # * machine options (including accelerator): in some architectures, CPU models # may look different depending on machine and accelerator options. (Except for # CPU models reported as "static" in query-cpu-definitions.) # * "-cpu" arguments and global properties: arguments to the -cpu option and # global properties may affect expansion of CPU models. Using # query-cpu-model-expansion while using these is not advised. # # Some architectures may not support all expansion types. s390x supports # "full" and "static". # # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is # not supported, if the model cannot be expanded, if the model contains # an unknown CPU definition name, unknown properties or properties # with a wrong type. Also returns an error if an expansion type is # not supported. # # Since: 2.8.0 ## { 'command': 'query-cpu-model-expansion', 'data': { 'type': 'CpuModelExpansionType', 'model': 'CpuModelInfo' }, 'returns': 'CpuModelExpansionInfo' } 4:46:25 PM ◄ qapi-schema.json </quotation> What do you think? Thank you. Marc [1] https://libvirt.org/formatdomain.html#elementsCPU -- Beste Grüße / Kind regards Marc Hartmayer IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On 20.10.2017 13:09, Marc Hartmayer wrote:
On Thu, Oct 12, 2017 at 02:07 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
On Mon, Oct 09, 2017 at 10:16:48 +0200, Marc Hartmayer wrote:
On Thu, Oct 05, 2017 at 02:11 PM +0200, Jiri Denemark <jdenemar@redhat.com> wrote:
But it's going to be a bit complicated because we ask QEMU what the host CPU is and the interface we used would need to be enhanced to give us different results for all supported machine types so that we can select the right one when a domain is started.
So how do we deal with this?
I think we need to start discussing this on qemu-devel list. If I remember correctly, David Hildenbrand is the original author of the query-cpu-model-expansion QMP command.
Please, Cc me so that the thread is more visible to me.
Thanks,
Jirka
Hi all,
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Yes, the host model may vary depending on QEMU version and to some degree also on compatibility machines (although I always try to push people to not introduce any new such stuff). Quoting for the libvirt documentation: https://libvirt.org/formatdomain.html "host-model The host-model mode is essentially a shortcut to copying host CPU definition from capabilities XML into domain XML. Since the CPU definition is copied just before starting a domain, exactly the same XML can be used on different hosts while still providing the best guest CPU each host supports." You're telling me that that copying does not happen, which seems to be the problematic part about this in my opinion. So I am not sure if probing anything else (as you noted below) is the correct thing to do. Copy it and you have a migration_safe and even static version of the _current_ host CPU.
For determining the actual host-model the QMP command 'query-cpu-model-expansion' is used. This is done once per QEMU binary and the result of it is cached by libvirt. The problem with that is that libvirt queries with the newest machine type of the QEMU binary for the host CPU model. But now we realized that we have to consider the machine type of each domain for the determination of the host CPU model of a domain.
We could now either probe the host CPU model for each QEMU binary + machine type combination and for this we've to start a new QEMU process each time. Or we can add a QMP command/parameter that allows us to determine the host CPU model(s) with respect to the passed machine type and/or all supported machine types.
The QMP command query-cpu-model-expansion is currently described in qapi-schema.json as follows:
<quotation> ## # @query-cpu-model-expansion: # # Expands a given CPU model (or a combination of CPU model + additional options) # to different granularities, allowing tooling to get an understanding what a # specific CPU model looks like in QEMU under a certain configuration. # # This interface can be used to query the "host" CPU model. # # The data returned by this command may be affected by: # # * QEMU version: CPU models may look different depending on the QEMU version. # (Except for CPU models reported as "static" in query-cpu-definitions.) # * machine-type: CPU model may look different depending on the machine-type. # (Except for CPU models reported as "static" in query-cpu-definitions.) # * machine options (including accelerator): in some architectures, CPU models # may look different depending on machine and accelerator options. (Except for # CPU models reported as "static" in query-cpu-definitions.) # * "-cpu" arguments and global properties: arguments to the -cpu option and # global properties may affect expansion of CPU models. Using # query-cpu-model-expansion while using these is not advised. # # Some architectures may not support all expansion types. s390x supports # "full" and "static". # # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is # not supported, if the model cannot be expanded, if the model contains # an unknown CPU definition name, unknown properties or properties # with a wrong type. Also returns an error if an expansion type is # not supported. # # Since: 2.8.0 ## { 'command': 'query-cpu-model-expansion', 'data': { 'type': 'CpuModelExpansionType', 'model': 'CpuModelInfo' }, 'returns': 'CpuModelExpansionInfo' }
4:46:25 PM ◄ qapi-schema.json </quotation>
What do you think?
Thank you.
Marc
[1] https://libvirt.org/formatdomain.html#elementsCPU
-- Beste Grüße / Kind regards Marc Hartmayer
IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
-- Thanks, David
On Fri, Oct 20, 2017 at 13:37:42 +0200, David Hildenbrand wrote:
On 20.10.2017 13:09, Marc Hartmayer wrote:
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Yes, the host model may vary depending on QEMU version and to some degree also on compatibility machines (although I always try to push people to not introduce any new such stuff).
Quoting for the libvirt documentation: https://libvirt.org/formatdomain.html
"host-model The host-model mode is essentially a shortcut to copying host CPU definition from capabilities XML into domain XML. Since the CPU definition is copied just before starting a domain, exactly the same XML can be used on different hosts while still providing the best guest CPU each host supports."
You're telling me that that copying does not happen, which seems to be the problematic part about this in my opinion.
So I am not sure if probing anything else (as you noted below) is the correct thing to do. Copy it and you have a migration_safe and even static version of the _current_ host CPU.
The thing is libvirt calls query-cpu-model-expansion to check what the host CPU is. This 'host-model' CPU is replaced with the probed CPU model when a domain starts. The problem described by Marc is that the probed CPU model cannot be used universally with all machine types. So starting a domain on such host with machine type s390-virtio-ccw-2.10 works, but a domain with machine type s390-virtio-ccw-2.9 fails to start with the same probed CPU model. Jirka
On 20.10.2017 14:50, Jiri Denemark wrote:
On Fri, Oct 20, 2017 at 13:37:42 +0200, David Hildenbrand wrote:
On 20.10.2017 13:09, Marc Hartmayer wrote:
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Yes, the host model may vary depending on QEMU version and to some degree also on compatibility machines (although I always try to push people to not introduce any new such stuff).
Quoting for the libvirt documentation: https://libvirt.org/formatdomain.html
"host-model The host-model mode is essentially a shortcut to copying host CPU definition from capabilities XML into domain XML. Since the CPU definition is copied just before starting a domain, exactly the same XML can be used on different hosts while still providing the best guest CPU each host supports."
You're telling me that that copying does not happen, which seems to be the problematic part about this in my opinion.
So I am not sure if probing anything else (as you noted below) is the correct thing to do. Copy it and you have a migration_safe and even static version of the _current_ host CPU.
The thing is libvirt calls query-cpu-model-expansion to check what the host CPU is. This 'host-model' CPU is replaced with the probed CPU model when a domain starts. The problem described by Marc is that the probed CPU model cannot be used universally with all machine types. So starting a domain on such host with machine type s390-virtio-ccw-2.10 works, but a domain with machine type s390-virtio-ccw-2.9 fails to start with the same probed CPU model.
My assumption would be that the CPU model is copied into the XML when the domain fist starts. This is what the documentation describes. So when upgrading QEMU, the CPU model in the XML is still the same (z13), even though something different is now reported in the host info after upgrading QEMU (z14). In this case it would continue to work. The problem is that the CPU model is not copied into the XML doesn't remain there, right? It is suddenly replaced with a z14 host model.
Jirka
-- Thanks, David
On Fri, Oct 20, 2017 at 15:04:57 +0200, David Hildenbrand wrote:
On 20.10.2017 14:50, Jiri Denemark wrote:
The thing is libvirt calls query-cpu-model-expansion to check what the host CPU is. This 'host-model' CPU is replaced with the probed CPU model when a domain starts. The problem described by Marc is that the probed CPU model cannot be used universally with all machine types. So starting a domain on such host with machine type s390-virtio-ccw-2.10 works, but a domain with machine type s390-virtio-ccw-2.9 fails to start with the same probed CPU model.
My assumption would be that the CPU model is copied into the XML when the domain fist starts. This is what the documentation describes.
The CPU model is copied into the XML each time the domain starts.
So when upgrading QEMU, the CPU model in the XML is still the same (z13), even though something different is now reported in the host info after upgrading QEMU (z14).
In this case it would continue to work.
The problem is that the CPU model is not copied into the XML doesn't remain there, right? It is suddenly replaced with a z14 host model.
Even preserving the actual CPU model in the XML from the first start wouldn't solve the issue. You can always create a new domain with s390-virtio-ccw-2.9 machine type even on the upgraded host. Jirka
On Fri, Oct 20, 2017 at 13:09:26 +0200, Marc Hartmayer wrote:
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
For determining the actual host-model the QMP command 'query-cpu-model-expansion' is used. This is done once per QEMU binary and the result of it is cached by libvirt. The problem with that is that libvirt queries with the newest machine type of the QEMU binary for the host CPU model.
No, libvirt probes QEMU with -machine none.
We could now either probe the host CPU model for each QEMU binary + machine type combination and for this we've to start a new QEMU process each time.
This is not really a viable solution. Jirka
Hi all,
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Actually, what is the cause of that problem? I assume it is the gs feature (gs_allowed)? We should really avoid such things (..._allowed) for CPU model features in the future and clue all new such stuff to cpumodel_allowed. -- Thanks, David
On 10/20/2017 03:16 PM, David Hildenbrand wrote:
Hi all,
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Actually, what is the cause of that problem? I assume it is the gs feature (gs_allowed)?
We should really avoid such things (..._allowed) for CPU model features in the future and clue all new such stuff to cpumodel_allowed.
Yes, starting a guest with <os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/> results in qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs Tying it to cpumodel_allowed would not help, migration-wise though. libvirt would still transform <os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/> into -cpu z14-base,aen=on,cmmnt=on,aefsi=on,mepoch=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on, \ msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,vxeh=on,vxpd=on, \ esop=on,iep=on,cte=on,ais=on,gs=on,zpci=on,sea_esop2=on,te=on,cmm=on ^^^^^ because cpu model is certainly there. Now the guest would start but migration would later fail because what we create now would never have been possible with 2.9. If libvirt could get information from QEMU depending on the machine version, this would make it work. But of course this has other issues. Christian
On 20.10.2017 15:36, Christian Borntraeger wrote:
On 10/20/2017 03:16 PM, David Hildenbrand wrote:
Hi all,
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Actually, what is the cause of that problem? I assume it is the gs feature (gs_allowed)?
We should really avoid such things (..._allowed) for CPU model features in the future and clue all new such stuff to cpumodel_allowed.
Yes, starting a guest with <os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
results in
qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs
Tying it to cpumodel_allowed would not help, migration-wise though. libvirt would still transform
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
My point was, that the host model would have to be copied and _remain_ there when s390-ccw-virtio was expanded to s390-ccw-virtio-2.9. So really replacing <cpu mode='host-model'/> by the model z13-base.... This would at least fix this issue. Just like s390-ccw-virtio get's replaced and remains thats way. But this might for sure have other problems.
into -cpu z14-base,aen=on,cmmnt=on,aefsi=on,mepoch=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on, \ msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,vxeh=on,vxpd=on, \ esop=on,iep=on,cte=on,ais=on,gs=on,zpci=on,sea_esop2=on,te=on,cmm=on ^^^^^ because cpu model is certainly there. Now the guest would start but migration would later fail because what we create now would never have been possible with 2.9.
Migration is a totally different story, as tooling has to make sure to find a CPU model that is compatible over all host models. So cpumodel_allowed would indeed work. (at least in my theory ;) )
If libvirt could get information from QEMU depending on the machine version, this would make it work. But of course this has other issues.
I am not sure if that is the right thing to do. The documentation states clearly that the host model is copied. If that is not runnable, fix the setup.
Christian
-- Thanks, David
On 10/20/2017 03:43 PM, David Hildenbrand wrote:
On 20.10.2017 15:36, Christian Borntraeger wrote:
On 10/20/2017 03:16 PM, David Hildenbrand wrote:
Hi all,
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Actually, what is the cause of that problem? I assume it is the gs feature (gs_allowed)?
We should really avoid such things (..._allowed) for CPU model features in the future and clue all new such stuff to cpumodel_allowed.
Yes, starting a guest with <os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
results in
qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs
Tying it to cpumodel_allowed would not help, migration-wise though. libvirt would still transform
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
My point was, that the host model would have to be copied and _remain_ there when s390-ccw-virtio was expanded to s390-ccw-virtio-2.9.
So really replacing <cpu mode='host-model'/> by the model z13-base.... This would at least fix this issue. Just like s390-ccw-virtio get's replaced and remains thats way.
But this might for sure have other problems.
The problem goes much further. A fresh guest with <os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/> does not start. No migration from an older system is necessary.
On 20.10.2017 15:49, Christian Borntraeger wrote:
On 10/20/2017 03:43 PM, David Hildenbrand wrote:
On 20.10.2017 15:36, Christian Borntraeger wrote:
On 10/20/2017 03:16 PM, David Hildenbrand wrote:
Hi all,
we recently encountered the problem that the 'host-model' [1] has to be related to the machine type of a domain. We have following problem:
Let's assume we've a z13 system with a QEMU 2.9 and we define a domain using the default s390-virtio-ccw machine together with the host-model CPU mode [1]. The definition will have the machine expanded to s390-virtio-ccw-2.9 but retain the host-model CPU mode in the domain definition. In a next step we upgrade to QEMU 2.10 (first version to recognize z14). Everything is still fine, even though the machine runs in 2.9 compatibility mode. Finally we upgrade to a z14. As a consequence it is not possible to start the domain anymore as the machine type doesn't support our CPU host model (which is expanded at start time of the domain).
Actually, what is the cause of that problem? I assume it is the gs feature (gs_allowed)?
We should really avoid such things (..._allowed) for CPU model features in the future and clue all new such stuff to cpumodel_allowed.
Yes, starting a guest with <os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
results in
qemu-system-s390x: Some features requested in the CPU model are not available in the configuration: gs
Tying it to cpumodel_allowed would not help, migration-wise though. libvirt would still transform
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
My point was, that the host model would have to be copied and _remain_ there when s390-ccw-virtio was expanded to s390-ccw-virtio-2.9.
So really replacing <cpu mode='host-model'/> by the model z13-base.... This would at least fix this issue. Just like s390-ccw-virtio get's replaced and remains thats way.
But this might for sure have other problems.
The problem goes much further. A fresh guest with
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
does not start. No migration from an older system is necessary.
Yes, as stated in the documentation "copying host CPU definition from capabilities XML" this can not work. And it works just as documented. Not saying this is a nice thing :) I think we should try to fix gs_allowed (if possible) and avoid something like that in the future. This would avoid other complexity involved when suddenly having X host models. -- Thanks, David
On 10/20/2017 03:51 PM, David Hildenbrand wrote: [...]
The problem goes much further. A fresh guest with
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
does not start. No migration from an older system is necessary.
Yes, as stated in the documentation "copying host CPU definition from capabilities XML" this can not work. And it works just as documented. Not saying this is a nice thing :)
I think we should try to fix gs_allowed (if possible) and avoid something like that in the future. This would avoid other complexity involved when suddenly having X host models.
Maybe this one is really a proper fix. It will allow the guest to start and on migration the cpu model will complain if the target cannot provide gs. Similar things can happen if - for example - the host kernel lacks some features. diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 77169bb..97a08fa 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data) s390mc->ri_allowed = true; s390mc->cpu_model_allowed = true; s390mc->css_migration_enabled = true; - s390mc->gs_allowed = true; mc->init = ccw_init; mc->reset = s390_machine_reset; mc->hot_add_cpu = s390_hot_add_cpu; @@ -509,12 +508,6 @@ bool cpu_model_allowed(void) return get_machine_class()->cpu_model_allowed; } -bool gs_allowed(void) -{ - /* for "none" machine this results in true */ - return get_machine_class()->gs_allowed; -} - static char *machine_get_loadparm(Object *obj, Error **errp) { S390CcwMachineState *ms = S390_CCW_MACHINE(obj); @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc) { S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc); - s390mc->gs_allowed = false; ccw_machine_2_10_class_options(mc); SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9); s390mc->css_migration_enabled = false; diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h index a9a90c2..1de53b0 100644 --- a/include/hw/s390x/s390-virtio-ccw.h +++ b/include/hw/s390x/s390-virtio-ccw.h @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass { bool ri_allowed; bool cpu_model_allowed; bool css_migration_enabled; - bool gs_allowed; } S390CcwMachineClass; /* runtime-instrumentation allowed by the machine */ diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c index a0d5052..3f13fc2 100644 --- a/target/s390x/kvm.c +++ b/target/s390x/kvm.c @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_ri = 1; } } - if (gs_allowed()) { + if (cpu_model_allowed()) { if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) { cap_gs = 1; }
On 20.10.2017 16:02, Christian Borntraeger wrote:
On 10/20/2017 03:51 PM, David Hildenbrand wrote: [...]
The problem goes much further. A fresh guest with
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
does not start. No migration from an older system is necessary.
Yes, as stated in the documentation "copying host CPU definition from capabilities XML" this can not work. And it works just as documented. Not saying this is a nice thing :)
I think we should try to fix gs_allowed (if possible) and avoid something like that in the future. This would avoid other complexity involved when suddenly having X host models.
Maybe this one is really a proper fix. It will allow the guest to start and on migration the cpu model will complain if the target cannot provide gs. Similar things can happen if - for example - the host kernel lacks some features.
Right, just what I had in mind.
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 77169bb..97a08fa 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data) s390mc->ri_allowed = true; s390mc->cpu_model_allowed = true; s390mc->css_migration_enabled = true; - s390mc->gs_allowed = true; mc->init = ccw_init; mc->reset = s390_machine_reset; mc->hot_add_cpu = s390_hot_add_cpu; @@ -509,12 +508,6 @@ bool cpu_model_allowed(void) return get_machine_class()->cpu_model_allowed; }
-bool gs_allowed(void) -{ - /* for "none" machine this results in true */ - return get_machine_class()->gs_allowed; -} - static char *machine_get_loadparm(Object *obj, Error **errp) { S390CcwMachineState *ms = S390_CCW_MACHINE(obj); @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc) { S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
- s390mc->gs_allowed = false; ccw_machine_2_10_class_options(mc); SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9); s390mc->css_migration_enabled = false; diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h index a9a90c2..1de53b0 100644 --- a/include/hw/s390x/s390-virtio-ccw.h +++ b/include/hw/s390x/s390-virtio-ccw.h @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass { bool ri_allowed; bool cpu_model_allowed; bool css_migration_enabled; - bool gs_allowed; } S390CcwMachineClass;
/* runtime-instrumentation allowed by the machine */ diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c index a0d5052..3f13fc2 100644 --- a/target/s390x/kvm.c +++ b/target/s390x/kvm.c @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_ri = 1; } } - if (gs_allowed()) { + if (cpu_model_allowed()) { if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) { cap_gs = 1; }
And the last hunk makes sure we keep same handling for machines without CPU model support and therefore no way to mask support. For all recent machines, we expect CPU model checks to be in place. Will have to think about this a bit more. Will you send this as a proper patch? -- Thanks, David
On 10/20/2017 04:06 PM, David Hildenbrand wrote:
On 20.10.2017 16:02, Christian Borntraeger wrote:
On 10/20/2017 03:51 PM, David Hildenbrand wrote: [...]
The problem goes much further. A fresh guest with
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
does not start. No migration from an older system is necessary.
Yes, as stated in the documentation "copying host CPU definition from capabilities XML" this can not work. And it works just as documented. Not saying this is a nice thing :)
I think we should try to fix gs_allowed (if possible) and avoid something like that in the future. This would avoid other complexity involved when suddenly having X host models.
Maybe this one is really a proper fix. It will allow the guest to start and on migration the cpu model will complain if the target cannot provide gs. Similar things can happen if - for example - the host kernel lacks some features.
Right, just what I had in mind.
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 77169bb..97a08fa 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data) s390mc->ri_allowed = true; s390mc->cpu_model_allowed = true; s390mc->css_migration_enabled = true; - s390mc->gs_allowed = true; mc->init = ccw_init; mc->reset = s390_machine_reset; mc->hot_add_cpu = s390_hot_add_cpu; @@ -509,12 +508,6 @@ bool cpu_model_allowed(void) return get_machine_class()->cpu_model_allowed; }
-bool gs_allowed(void) -{ - /* for "none" machine this results in true */ - return get_machine_class()->gs_allowed; -} - static char *machine_get_loadparm(Object *obj, Error **errp) { S390CcwMachineState *ms = S390_CCW_MACHINE(obj); @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc) { S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
- s390mc->gs_allowed = false; ccw_machine_2_10_class_options(mc); SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9); s390mc->css_migration_enabled = false; diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h index a9a90c2..1de53b0 100644 --- a/include/hw/s390x/s390-virtio-ccw.h +++ b/include/hw/s390x/s390-virtio-ccw.h @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass { bool ri_allowed; bool cpu_model_allowed; bool css_migration_enabled; - bool gs_allowed; } S390CcwMachineClass;
/* runtime-instrumentation allowed by the machine */ diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c index a0d5052..3f13fc2 100644 --- a/target/s390x/kvm.c +++ b/target/s390x/kvm.c @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_ri = 1; } } - if (gs_allowed()) { + if (cpu_model_allowed()) { if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) { cap_gs = 1; }
And the last hunk makes sure we keep same handling for machines without CPU model support and therefore no way to mask support. For all recent machines, we expect CPU model checks to be in place.
Will have to think about this a bit more. Will you send this as a proper patch?
After thinking about it :-)
On 10/20/2017 04:12 PM, Christian Borntraeger wrote:
On 10/20/2017 04:06 PM, David Hildenbrand wrote:
On 20.10.2017 16:02, Christian Borntraeger wrote:
On 10/20/2017 03:51 PM, David Hildenbrand wrote: [...]
The problem goes much further. A fresh guest with
<os> <type arch='s390x' machine='s390-ccw-virtio-2.9'>hvm</type> </os> <cpu mode='host-model'/>
does not start. No migration from an older system is necessary.
Yes, as stated in the documentation "copying host CPU definition from capabilities XML" this can not work. And it works just as documented. Not saying this is a nice thing :)
I think we should try to fix gs_allowed (if possible) and avoid something like that in the future. This would avoid other complexity involved when suddenly having X host models.
Maybe this one is really a proper fix. It will allow the guest to start and on migration the cpu model will complain if the target cannot provide gs. Similar things can happen if - for example - the host kernel lacks some features.
Right, just what I had in mind.
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 77169bb..97a08fa 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -430,7 +430,6 @@ static void ccw_machine_class_init(ObjectClass *oc, void *data) s390mc->ri_allowed = true; s390mc->cpu_model_allowed = true; s390mc->css_migration_enabled = true; - s390mc->gs_allowed = true; mc->init = ccw_init; mc->reset = s390_machine_reset; mc->hot_add_cpu = s390_hot_add_cpu; @@ -509,12 +508,6 @@ bool cpu_model_allowed(void) return get_machine_class()->cpu_model_allowed; }
-bool gs_allowed(void) -{ - /* for "none" machine this results in true */ - return get_machine_class()->gs_allowed; -} - static char *machine_get_loadparm(Object *obj, Error **errp) { S390CcwMachineState *ms = S390_CCW_MACHINE(obj); @@ -757,7 +750,6 @@ static void ccw_machine_2_9_class_options(MachineClass *mc) { S390CcwMachineClass *s390mc = S390_MACHINE_CLASS(mc);
- s390mc->gs_allowed = false; ccw_machine_2_10_class_options(mc); SET_MACHINE_COMPAT(mc, CCW_COMPAT_2_9); s390mc->css_migration_enabled = false; diff --git a/include/hw/s390x/s390-virtio-ccw.h b/include/hw/s390x/s390-virtio-ccw.h index a9a90c2..1de53b0 100644 --- a/include/hw/s390x/s390-virtio-ccw.h +++ b/include/hw/s390x/s390-virtio-ccw.h @@ -40,7 +40,6 @@ typedef struct S390CcwMachineClass { bool ri_allowed; bool cpu_model_allowed; bool css_migration_enabled; - bool gs_allowed; } S390CcwMachineClass;
/* runtime-instrumentation allowed by the machine */ diff --git a/target/s390x/kvm.c b/target/s390x/kvm.c index a0d5052..3f13fc2 100644 --- a/target/s390x/kvm.c +++ b/target/s390x/kvm.c @@ -362,7 +362,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_ri = 1; } } - if (gs_allowed()) { + if (cpu_model_allowed()) { if (kvm_vm_enable_cap(s, KVM_CAP_S390_GS, 0) == 0) { cap_gs = 1; }
And the last hunk makes sure we keep same handling for machines without CPU model support and therefore no way to mask support. For all recent machines, we expect CPU model checks to be in place.
Will have to think about this a bit more. Will you send this as a proper patch?
After thinking about it :-)
I intend to put some brain-power in this too. Probably next week. My general impression is, that I have a at places different understanding of how things should work compared to David. Especially when it comes to this concept of persistent copying, and also an end-user-digestible definition of what host-model does. (I think this with copying capabilities from whatever xml which is subject to convoluted caching is a bit too hard to digest for an end user not involved in the development of qemu and libvirt). I had a conversation with Boris a couple of hours ago, and it seems, things are pretty convoluted. If I understand the train of thought here (David) it can be summarized like this: For a certain QEMU binary no aspect of the cpu-models may depend on the machine type. In particular, compat properties and compat handling is not alowed to alter cpu-models (whether the available cpu models nor the capabilities of each of these). This we would have to make a part of the external interface. That way one could be sure that the 'cpu capabilities' are machine-type independent (that is, the same for all the machine types). Or did I get this completely wrong? (Based on the answer branches my think). Halil
I intend to put some brain-power in this too. Probably next week.
My general impression is, that I have a at places different understanding of how things should work compared to David. Especially when it comes to this concept of persistent copying, and also an end-user-digestible definition of what host-model does. (I think this with copying capabilities from whatever xml which is subject to convoluted caching is a bit too hard to digest for an end user not involved in the development of qemu and libvirt).
When reading the doc I was assuming that it would be a persistent copy. But it would only fix part of the issue. In the end, it specifies quite clearly what is copied. And if that is not runnable with the selected machine, bad luck. Therefore ...
I had a conversation with Boris a couple of hours ago, and it seems, things are pretty convoluted.
If I understand the train of thought here (David) it can be summarized like this: For a certain QEMU binary no aspect of the cpu-models may depend on the machine type. In particular, compat properties and compat handling is not alowed to alter cpu-models (whether the available cpu models nor the capabilities of each of these).
... I always recommended avoiding such compatibility settings in the machine. But before we had CPU models it was really all we had. No we should let go of it. We might be able to fix this one (gs) and take care of it in the future, but ...
This we would have to make a part of the external interface. That way one could be sure that the 'cpu capabilities' are machine-type independent (that is, the same for all the machine types).
... the problem is once nasty stuff like zPCI comes in place. If we ever have (other?) machine dependent stuff that toggles CPU features, we can only try limit the damage.
Or did I get this completely wrong? (Based on the answer branches my think).
Not at all.
Halil
-- Thanks, David
participants (5)
-
Christian Borntraeger -
David Hildenbrand -
Halil Pasic -
Jiri Denemark -
Marc Hartmayer