On Fri, Nov 01, 2019 at 06:43:16PM +0100, Christian Borntraeger wrote: > On the KVM forum I have discussed the default cpu model mode on s390. > Right now if the xml does not specify anything, libvirt defaults to > not specifying anything on the qemu command line (no -cpu statement) > which is the equivalent of -cpu host for s390 which is equivalent to > host-passthrough. While this enables all features it does not provide > any migration safety by default. > > So in fact we are kind of "broken" right now when it comes to safery. > > So we discussed that it would make sense that an empty xml should actually > be defaulted to host-model, which results in - as of today - the same guest > features but in a migration safe way. > > There is another change planned right now to actually make the cpu model > present in an xml if none was specified. So we could actually do this change > before, together or after te other. Jiri and I think it probably makes most > sense to have both changes at the same time (in terms of libvirt version). > > Does anyone see an issue with changing the default model mode to "host-model" > if the xml does not specify anything else? Changing from "host-passthrough" to "host-model" is not a huge difference, but it is none the less a guest ABI change. "host-passthrough" doesn't provide migration safety in the face of differing hardware, it should still be valid for people with homogeneous hardware. So changing the model will potentially break some existing usage.

As the top priority we should definitely make sure that the guest XML gets updated to list "host-passthrough" when no CPU is specified, so that it reflects the current reality. If libvirt gets this information

from QEMU, then at some point down the line we can potentially change the default by tieing a new default to a versioned machine type. We have to wait a little while for old libvirt's to become irrelevant wrt new QEMU though. This is the same issue with changing the default CPU on x86 which, though that's possibly harder on x86 as the scope of any change is more significant.

On 02.11.19 11:32, Daniel P. Berrangé wrote: > On Fri, Nov 01, 2019 at 06:43:16PM +0100, Christian Borntraeger wrote: > > On the KVM forum I have discussed the default cpu model mode on s390. > > Right now if the xml does not specify anything, libvirt defaults to > > not specifying anything on the qemu command line (no -cpu statement) > > which is the equivalent of -cpu host for s390 which is equivalent to > > host-passthrough. While this enables all features it does not provide > > any migration safety by default. > > > > So in fact we are kind of "broken" right now when it comes to safery. > > > > So we discussed that it would make sense that an empty xml should actually > > be defaulted to host-model, which results in - as of today - the same guest > > features but in a migration safe way. > > > > There is another change planned right now to actually make the cpu model > > present in an xml if none was specified. So we could actually do this change > > before, together or after te other. Jiri and I think it probably makes most > > sense to have both changes at the same time (in terms of libvirt version). > > > > Does anyone see an issue with changing the default model mode to "host-model" > > if the xml does not specify anything else? > > Changing from "host-passthrough" to "host-model" is not a huge difference, > but it is none the less a guest ABI change. "host-passthrough" doesn't > provide migration safety in the face of differing hardware, it should still > be valid for people with homogeneous hardware. So changing the model will > potentially break some existing usage. I guess on s390x this is not the case ("-cpu host", no "-cpu", and passing the expanded "host" model will result in the same guest ABI, in contrast to x86 AFAIK). There is this special case, though, where we have old QEMUs without CPU model support. Not sure how to deal with that, then.

On 08.11.19 12:43, Daniel P. Berrangé wrote: > On Mon, Nov 04, 2019 at 11:49:01AM +0100, David Hildenbrand wrote: >> On 02.11.19 11:32, Daniel P. Berrangé wrote: >>> On Fri, Nov 01, 2019 at 06:43:16PM +0100, Christian Borntraeger wrote: >>>> On the KVM forum I have discussed the default cpu model mode on s390. >>>> Right now if the xml does not specify anything, libvirt defaults to >>>> not specifying anything on the qemu command line (no -cpu statement) >>>> which is the equivalent of -cpu host for s390 which is equivalent to >>>> host-passthrough. While this enables all features it does not provide >>>> any migration safety by default. >>>> >>>> So in fact we are kind of "broken" right now when it comes to safery. >>>> >>>> So we discussed that it would make sense that an empty xml should actually >>>> be defaulted to host-model, which results in - as of today - the same guest >>>> features but in a migration safe way. >>>> >>>> There is another change planned right now to actually make the cpu model >>>> present in an xml if none was specified. So we could actually do this change >>>> before, together or after te other. Jiri and I think it probably makes most >>>> sense to have both changes at the same time (in terms of libvirt version). >>>> >>>> Does anyone see an issue with changing the default model mode to "host-model" >>>> if the xml does not specify anything else? >>> >>> Changing from "host-passthrough" to "host-model" is not a huge difference, >>> but it is none the less a guest ABI change. "host-passthrough" doesn't >>> provide migration safety in the face of differing hardware, it should still >>> be valid for people with homogeneous hardware. So changing the model will >>> potentially break some existing usage. >> >> I guess on s390x this is not the case ("-cpu host", no "-cpu", and passing >> the expanded "host" model will result in the same guest ABI, in contrast to >> x86 AFAIK). There is this special case, though, where we have old QEMUs >> without CPU model support. Not sure how to deal with that, then. > > I'm still not sure I understand the s390 CPU ABI rules. > > Current libvirt, no <cpu>, and thus no -cpu. > > IIUC this is functionally identical to using "-cpu host" and/or > <cpu mode="host-passthrough"/> > > If you are using "-cpu host" / <cpu mode="host-passthrough"> can you > live migrate to another host with identical physical CPUs + firmware ? > > > Assuming this is possible, then, can you live migrate a QEMU guest > booted with <cpu mode="host-passthrough">, to a QEMU guest booted > with <cpu mode="host-model"> ? Not sure I understand your question. With "can", do you mean "the guest has the same guest visible CPU features and types"?

On 08.11.19 12:52, Daniel P. Berrangé wrote: > On Fri, Nov 08, 2019 at 12:49:23PM +0100, Christian Borntraeger wrote: >> >> >> On 08.11.19 12:43, Daniel P. Berrangé wrote: >>> On Mon, Nov 04, 2019 at 11:49:01AM +0100, David Hildenbrand wrote: >>>> On 02.11.19 11:32, Daniel P. Berrangé wrote: >>>>> On Fri, Nov 01, 2019 at 06:43:16PM +0100, Christian Borntraeger wrote: >>>>>> On the KVM forum I have discussed the default cpu model mode on s390. >>>>>> Right now if the xml does not specify anything, libvirt defaults to >>>>>> not specifying anything on the qemu command line (no -cpu statement) >>>>>> which is the equivalent of -cpu host for s390 which is equivalent to >>>>>> host-passthrough. While this enables all features it does not provide >>>>>> any migration safety by default. >>>>>> >>>>>> So in fact we are kind of "broken" right now when it comes to safery. >>>>>> >>>>>> So we discussed that it would make sense that an empty xml should actually >>>>>> be defaulted to host-model, which results in - as of today - the same guest >>>>>> features but in a migration safe way. >>>>>> >>>>>> There is another change planned right now to actually make the cpu model >>>>>> present in an xml if none was specified. So we could actually do this change >>>>>> before, together or after te other. Jiri and I think it probably makes most >>>>>> sense to have both changes at the same time (in terms of libvirt version). >>>>>> >>>>>> Does anyone see an issue with changing the default model mode to "host-model" >>>>>> if the xml does not specify anything else? >>>>> >>>>> Changing from "host-passthrough" to "host-model" is not a huge difference, >>>>> but it is none the less a guest ABI change. "host-passthrough" doesn't >>>>> provide migration safety in the face of differing hardware, it should still >>>>> be valid for people with homogeneous hardware. So changing the model will >>>>> potentially break some existing usage. >>>> >>>> I guess on s390x this is not the case ("-cpu host", no "-cpu", and passing >>>> the expanded "host" model will result in the same guest ABI, in contrast to >>>> x86 AFAIK). There is this special case, though, where we have old QEMUs >>>> without CPU model support. Not sure how to deal with that, then. >>> >>> I'm still not sure I understand the s390 CPU ABI rules. >>> >>> Current libvirt, no <cpu>, and thus no -cpu. >>> >>> IIUC this is functionally identical to using "-cpu host" and/or >>> <cpu mode="host-passthrough"/> >>> >>> If you are using "-cpu host" / <cpu mode="host-passthrough"> can you >>> live migrate to another host with identical physical CPUs + firmware ? >>> >>> >>> Assuming this is possible, then, can you live migrate a QEMU guest >>> booted with <cpu mode="host-passthrough">, to a QEMU guest booted >>> with <cpu mode="host-model"> ? >> >> Not sure I understand your question. With "can", do you mean "the guest >> has the same guest visible CPU features and types"? > > Yes, I mean the migration should succeed from QEMU's POV and additionally > the guest OS should not see any change in CPU ABI exposed from the host. The guest ABI is the same and migration also seems to work. I think your concern boils down to the case that source and target have a different libvirt version (but qemu and kernel and firmware and hardware are identical). So for that case this change would break things if host-model and host-passthrough are not identical. So as of today we have no concern. Now: Would it be a concern if a future QEMU decides to change that equivalence somehow?

On 08.11.19 14:10, Daniel P. Berrangé wrote: > On Fri, Nov 08, 2019 at 01:56:47PM +0100, Christian Borntraeger wrote: >> >> >> On 08.11.19 12:52, Daniel P. Berrangé wrote: >>> On Fri, Nov 08, 2019 at 12:49:23PM +0100, Christian Borntraeger wrote: >>>> >>>> >>>> On 08.11.19 12:43, Daniel P. Berrangé wrote: >>>>> On Mon, Nov 04, 2019 at 11:49:01AM +0100, David Hildenbrand wrote: >>>>>> On 02.11.19 11:32, Daniel P. Berrangé wrote: >>>>>>> On Fri, Nov 01, 2019 at 06:43:16PM +0100, Christian Borntraeger wrote: >>>>>>>> On the KVM forum I have discussed the default cpu model mode on s390. >>>>>>>> Right now if the xml does not specify anything, libvirt defaults to >>>>>>>> not specifying anything on the qemu command line (no -cpu statement) >>>>>>>> which is the equivalent of -cpu host for s390 which is equivalent to >>>>>>>> host-passthrough. While this enables all features it does not provide >>>>>>>> any migration safety by default. >>>>>>>> >>>>>>>> So in fact we are kind of "broken" right now when it comes to safery. >>>>>>>> >>>>>>>> So we discussed that it would make sense that an empty xml should actually >>>>>>>> be defaulted to host-model, which results in - as of today - the same guest >>>>>>>> features but in a migration safe way. >>>>>>>> >>>>>>>> There is another change planned right now to actually make the cpu model >>>>>>>> present in an xml if none was specified. So we could actually do this change >>>>>>>> before, together or after te other. Jiri and I think it probably makes most >>>>>>>> sense to have both changes at the same time (in terms of libvirt version). >>>>>>>> >>>>>>>> Does anyone see an issue with changing the default model mode to "host-model" >>>>>>>> if the xml does not specify anything else? >>>>>>> >>>>>>> Changing from "host-passthrough" to "host-model" is not a huge difference, >>>>>>> but it is none the less a guest ABI change. "host-passthrough" doesn't >>>>>>> provide migration safety in the face of differing hardware, it should still >>>>>>> be valid for people with homogeneous hardware. So changing the model will >>>>>>> potentially break some existing usage. >>>>>> >>>>>> I guess on s390x this is not the case ("-cpu host", no "-cpu", and passing >>>>>> the expanded "host" model will result in the same guest ABI, in contrast to >>>>>> x86 AFAIK). There is this special case, though, where we have old QEMUs >>>>>> without CPU model support. Not sure how to deal with that, then. >>>>> >>>>> I'm still not sure I understand the s390 CPU ABI rules. >>>>> >>>>> Current libvirt, no <cpu>, and thus no -cpu. >>>>> >>>>> IIUC this is functionally identical to using "-cpu host" and/or >>>>> <cpu mode="host-passthrough"/> >>>>> >>>>> If you are using "-cpu host" / <cpu mode="host-passthrough"> can you >>>>> live migrate to another host with identical physical CPUs + firmware ? >>>>> >>>>> >>>>> Assuming this is possible, then, can you live migrate a QEMU guest >>>>> booted with <cpu mode="host-passthrough">, to a QEMU guest booted >>>>> with <cpu mode="host-model"> ? >>>> >>>> Not sure I understand your question. With "can", do you mean "the guest >>>> has the same guest visible CPU features and types"? >>> >>> Yes, I mean the migration should succeed from QEMU's POV and additionally >>> the guest OS should not see any change in CPU ABI exposed from the host. >> >> The guest ABI is the same and migration also seems to work. >> I think your concern boils down to the case that source and target >> have a different libvirt version (but qemu and kernel and firmware >> and hardware are identical). So for that case this change would break >> things if host-model and host-passthrough are not identical. >> So as of today we have no concern. >> >> Now: Would it be a concern if a future QEMU decides to change that >> equivalence somehow? > > If they're the same guest ABI, then what's the benefit in changing the > default to "host-model" instead of just continuing with "host-passthrough". > It seems like we're fine to just carry on with "host-passthrough" as > the default and insulate ourselves from any future risk of change. The benefit is that that todays default is not migration safe and users will find that out by random guest crashes if any of the parameters (CPU, kernel, qemu, firmware) is different. So really, todays default is just completely broken on s390 and thats why I want to change it. host-model instead is expanded by libvirt and the migration will be rejected if the target is incompatible (qemu will reject to run).