[libvirt] [PATCH] docs: formatdomain: Specify the default value of 'check' attribute
The 'check' attribute is referring to this: <cpu mode='custom' match='exact' check='full'> Upstream documentation says, it is: used to request a specific way of checking whether the virtual CPU matches the specification. It is usually safe to omit this attribute when starting a domain and stick with the default value. But doesn't tell *what* the default value is. It is check='partial'. Mention it so. Thanks: Jiri Denemark --- docs/formatdomain.html.in | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index d272cc1ba..8985da836 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -1287,10 +1287,11 @@ attribute can be used to request a specific way of checking whether the virtual CPU matches the specification. It is usually safe to omit this attribute when starting a domain and stick with the default - value. Once the domain starts, libvirt will automatically change the - <code>check</code> attribute to the best supported value to ensure the - virtual CPU does not change when the domain is migrated to another - host. The following values can be used: + value (<code>partial>/code>). Once the domain starts, libvirt will + automatically change the <code>check</code> attribute to the best + supported value to ensure the virtual CPU does not change when the + domain is migrated to another host. The following values can be + used: <dl> <dt><code>none</code></dt> @@ -1303,7 +1304,7 @@ <dt><code>partial</code></dt> <dd>Libvirt will check the guest CPU specification before starting a domain, but the rest is left on the hypervisor. It can still - provide a different virtual CPU.</dd> + provide a different virtual CPU. (This is the default.)</dd> <dt><code>full</code></dt> <dd>The virtual CPU created by the hypervisor will be checked -- 2.13.6
On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
The 'check' attribute is referring to this:
<cpu mode='custom' match='exact' check='full'>
Upstream documentation says, it is:
used to request a specific way of checking whether the virtual CPU matches the specification. It is usually safe to omit this attribute when starting a domain and stick with the default value.
But doesn't tell *what* the default value is. It is check='partial'. Mention it so.
Thanks: Jiri Denemark --- docs/formatdomain.html.in | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
NACK As I said on IRC, the default differs with guest architecture ('partial' for x86_64, 'none' for all others). Moreover this only applies to QEMU driver. Oh and it even changes based on the mode attribute to make not make it easy. But this is really an implementation detail. Libvirt will just use a suitable default to stay compatible with older releases. Jirka
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
[...]
But doesn't tell *what* the default value is. It is check='partial'. Mention it so.
[...]
NACK
As I said on IRC, the default differs with guest architecture ('partial' for x86_64, 'none' for all others).
Yep, I missed to distinguish that detail.
Moreover this only applies to QEMU driver. Oh and it even changes based on the mode attribute to make not make it easy. But this is really an implementation detail. Libvirt will just use a suitable default to stay compatible with older releases.
That's all good. If you think nothing needs to be clarified here, that's OK. (Some admins who operate a lot of VMs with libvirt will disagree with you, though.) And as we saw on IRC, `virt-install` muddies the water in this case by gratuitously adding a CPU parameter with check='full' (even if a user didn't explicitly specify it) when you simply import an image and look at the run-time guest XML: $> virt-install --name vm1 --ram 2048 \ --disk path=./vm1.qcow2,format=qcow2 --nographics \ --import --os-variant fedora27 $> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='full'> (You might, fairly, argue here that: "Well, that's a bug in `virt-install`, go complain there.") It all comes across as messy to an unsuspecting admin. (If a competent admin finds it confusing, good luck to new admins who want to wrap their head around this.) - - - Background for others reading: The admin who reported this was confused when he was creating guests with `virt-install`, which adds check='full' (as noted earlier), and the guest throws: error: Failed to start domain foo.org error: operation failed: guest CPU doesn't match specification: extra features: vme,arat So "somehow" QEMU added the CPU features 'vme' and 'arat' by itself, now you have to specify them in libvirt. So the admin ended up with a `sed` one-liner that updates the guest XML with the missing features: sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-" Versions: libvirt 3.2 and QEMU 2.9
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
[...]
But doesn't tell *what* the default value is. It is check='partial'. Mention it so.
[...]
NACK
As I said on IRC, the default differs with guest architecture ('partial' for x86_64, 'none' for all others).
Yep, I missed to distinguish that detail.
Moreover this only applies to QEMU driver. Oh and it even changes based on the mode attribute to make not make it easy. But this is really an implementation detail. Libvirt will just use a suitable default to stay compatible with older releases.
That's all good. If you think nothing needs to be clarified here, that's OK. (Some admins who operate a lot of VMs with libvirt will disagree with you, though.)
I'm not saying there's nothing to be clarified. Just that explicitly specifying the default value will not help at all since it's complicated and confusing by itself especially because of backward compatibility.
And as we saw on IRC, `virt-install` muddies the water in this case by gratuitously adding a CPU parameter with check='full' (even if a user didn't explicitly specify it) when you simply import an image and look at the run-time guest XML:
$> virt-install --name vm1 --ram 2048 \ --disk path=./vm1.qcow2,format=qcow2 --nographics \ --import --os-variant fedora27
$> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='full'>
(You might, fairly, argue here that: "Well, that's a bug in `virt-install`, go complain there.")
I guess it's just the default behavior of virt-install and it automatically passes the host CPU model to domain XML (without check attribute, relying on the default) when creating new domains. After all, any CPU model is going to be better than the default qemu64.
It all comes across as messy to an unsuspecting admin. (If a competent admin finds it confusing, good luck to new admins who want to wrap their head around this.)
- - -
Background for others reading: The admin who reported this was confused when he was creating guests with `virt-install`, which adds check='full' (as noted earlier), and the guest throws:
error: Failed to start domain foo.org error: operation failed: guest CPU doesn't match specification: extra features: vme,arat
If this really happens when creating a domain with virt-install, then it's clearly a bug. But I don't believe it's what happened. The dumpxml above says check='full', which means the domain was successfully started and its CPU def was updated according to QEMU and thus the check attribute was changed to 'full'. The vme and arat features are added because libvirt's and QEMU's vision of the particual CPU model differs. Libvirt specifies it without vme and arat while QEMU has them both included in the CPU model. Thus when libvirt asks for that CPU model, QEMU enables these extra features too and libvirt adds them to the domain XML so that it can make sure they don't disappear when the domain is migrated or save/restored.
So "somehow" QEMU added the CPU features 'vme' and 'arat' by itself, now you have to specify them in libvirt. So the admin ended up with a `sed` one-liner that updates the guest XML with the missing features:
sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear reason. It would be nice to finally see what the admin wanted to achieve, what steps they did, and what result they saw.
Versions: libvirt 3.2 and QEMU 2.9
It's certainly possible there was a bug in 3.2 which got fixed since then (there were several of them), but it's impossible to guess without seeing what they're doing. Jirka
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
[...]
But doesn't tell *what* the default value is. It is check='partial'. Mention it so.
[...]
NACK
As I said on IRC, the default differs with guest architecture ('partial' for x86_64, 'none' for all others).
Yep, I missed to distinguish that detail.
Moreover this only applies to QEMU driver. Oh and it even changes based on the mode attribute to make not make it easy. But this is really an implementation detail. Libvirt will just use a suitable default to stay compatible with older releases.
That's all good. If you think nothing needs to be clarified here, that's OK. (Some admins who operate a lot of VMs with libvirt will disagree with you, though.)
I'm not saying there's nothing to be clarified. Just that explicitly specifying the default value will not help at all since it's complicated and confusing by itself especially because of backward compatibility.
And as we saw on IRC, `virt-install` muddies the water in this case by gratuitously adding a CPU parameter with check='full' (even if a user didn't explicitly specify it) when you simply import an image and look at the run-time guest XML:
$> virt-install --name vm1 --ram 2048 \ --disk path=./vm1.qcow2,format=qcow2 --nographics \ --import --os-variant fedora27
$> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='full'>
(You might, fairly, argue here that: "Well, that's a bug in `virt-install`, go complain there.")
I guess it's just the default behavior of virt-install and it automatically passes the host CPU model to domain XML (without check attribute, relying on the default) when creating new domains. After all, any CPU model is going to be better than the default qemu64.
One additional note for that virt-install command, this is the cpu part of XML definition created by virt-install: <cpu mode="custom" match="exact"> <model>Broadwell</model> </cpu> The cpu model will be different based on the host HW, but as you can see, there is no "check='full'. You can verify it by appending "--dry-run --print-xml" to your virt-install command which will do nothing and only print the XML definition.
It all comes across as messy to an unsuspecting admin. (If a competent admin finds it confusing, good luck to new admins who want to wrap their head around this.)
- - -
Background for others reading: The admin who reported this was confused when he was creating guests with `virt-install`, which adds check='full' (as noted earlier), and the guest throws:
error: Failed to start domain foo.org error: operation failed: guest CPU doesn't match specification: extra features: vme,arat
If this really happens when creating a domain with virt-install, then it's clearly a bug. But I don't believe it's what happened. The dumpxml above says check='full', which means the domain was successfully started and its CPU def was updated according to QEMU and thus the check attribute was changed to 'full'.
The vme and arat features are added because libvirt's and QEMU's vision of the particual CPU model differs. Libvirt specifies it without vme and arat while QEMU has them both included in the CPU model. Thus when libvirt asks for that CPU model, QEMU enables these extra features too and libvirt adds them to the domain XML so that it can make sure they don't disappear when the domain is migrated or save/restored.
So "somehow" QEMU added the CPU features 'vme' and 'arat' by itself, now you have to specify them in libvirt. So the admin ended up with a `sed` one-liner that updates the guest XML with the missing features:
sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear reason. It would be nice to finally see what the admin wanted to achieve, what steps they did, and what result they saw.
Agreed, this is a mess. Just a note, if you need to edit XML and you would like to automate it, you can use virt-xml: virt-xml $domain --edit --cpu require=vme,require=arat Anyway, it shouldn't be needed to modify the XML in the first place. Pavel
On Wed, Jan 10, 2018 at 09:51:19AM +0100, Pavel Hrdina wrote:
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
[...]
One additional note for that virt-install command, this is the cpu part of XML definition created by virt-install:
<cpu mode="custom" match="exact"> <model>Broadwell</model> </cpu>
The cpu model will be different based on the host HW, but as you can see, there is no "check='full'.
Noted. The admin has noticed the difference between the offline and run-time definitions
You can verify it by appending "--dry-run --print-xml" to your virt-install command which will do nothing and only print the XML definition.
Yep, this too the admin is aware. Aside: is it expected that the run-time and offline config is intentionaly different for the 'check' attribute? Run-time: $> virsh start vm1 $> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='full'> ^^^^ Offline: $> virsh destroy vm1 $> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='partial'> ^^^^^^^ (Tried with: libvirt-daemon-kvm-3.7.0-2 and qemu-system-x86-2.10.0-4.) [...]
sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear reason. It would be nice to finally see what the admin wanted to achieve, what steps they did, and what result they saw.
Agreed, this is a mess.
No, the XML updating is not the "mess" (that was required in his case). The mess is how all this is subtly opaque to the admin.
Just a note, if you need to edit XML and you would like to automate it, you can use virt-xml:
virt-xml $domain --edit --cpu require=vme,require=arat
Yes, I already pointed that to the admin when he showed the `sed` variant.
Anyway, it shouldn't be needed to modify the XML in the first place.
It _was_ needed in his case (where updating QEMU broke). See my other reply to Jiri. -- /kashyap
On Wed, Jan 10, 2018 at 11:13:28 +0100, Kashyap Chamarthy wrote:
On Wed, Jan 10, 2018 at 09:51:19AM +0100, Pavel Hrdina wrote:
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
[...]
One additional note for that virt-install command, this is the cpu part of XML definition created by virt-install:
<cpu mode="custom" match="exact"> <model>Broadwell</model> </cpu>
The cpu model will be different based on the host HW, but as you can see, there is no "check='full'.
Noted. The admin has noticed the difference between the offline and run-time definitions
You can verify it by appending "--dry-run --print-xml" to your virt-install command which will do nothing and only print the XML definition.
Yep, this too the admin is aware.
Aside: is it expected that the run-time and offline config is intentionaly different for the 'check' attribute?
Yes.
Run-time:
$> virsh start vm1 $> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='full'> ^^^^
This will make sure that the CPU will not change at all (no features added or removed) when the running domain is migrated, save/restored, or snapshotted and reverted.
Offline:
$> virsh destroy vm1 $> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='partial'>
This is needed for backward compatibility (old libvirt allowed you to ask for some features if they were present in host CPU even though QEMU/KVM did not support them) and to mitigate the differences between libvirt's and QEMU's versions of CPU models (which may even differ based on machine type). Once the domain is started, libvirt notes all the small differences in the XML and changes check to 'full'.
(Tried with: libvirt-daemon-kvm-3.7.0-2 and qemu-system-x86-2.10.0-4.)
[...]
sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear reason. It would be nice to finally see what the admin wanted to achieve, what steps they did, and what result they saw.
Agreed, this is a mess.
No, the XML updating is not the "mess" (that was required in his case). The mess is how all this is subtly opaque to the admin.
Just a note, if you need to edit XML and you would like to automate it, you can use virt-xml:
virt-xml $domain --edit --cpu require=vme,require=arat
Yes, I already pointed that to the admin when he showed the `sed` variant.
Anyway, it shouldn't be needed to modify the XML in the first place.
It _was_ needed in his case (where updating QEMU broke).
It shouldn't be needed in any case though. It's either a bug or something wrong they did before. Jirka
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
[...]
I'm not saying there's nothing to be clarified. Just that explicitly specifying the default value will not help at all since it's complicated and confusing by itself especially because of backward compatibility.
Yeah, that I agree with you. It was just a v1 :-) (It would've been clearer if youd've said: "NACK — needs some fixing". [...]
$> virt-install --name vm1 --ram 2048 \ --disk path=./vm1.qcow2,format=qcow2 --nographics \ --import --os-variant fedora27
$> virsh dumpxml vm1 | grep check <cpu mode='custom' match='exact' check='full'>
(You might, fairly, argue here that: "Well, that's a bug in `virt-install`, go complain there.")
I guess it's just the default behavior of virt-install and it automatically passes the host CPU model to domain XML (without check attribute, relying on the default) when creating new domains. After all, any CPU model is going to be better than the default qemu64.
Right.
Background for others reading: The admin who reported this was confused when he was creating guests with `virt-install`, which adds check='full' (as noted earlier), and the guest throws:
error: Failed to start domain foo.org error: operation failed: guest CPU doesn't match specification: extra features: vme,arat
If this really happens when creating a domain with virt-install,
Right, it doesn't happen on domain start (I worded it poorly). The error occurred only after QEMU was updated (from 2.6.5 to 2.9).
then it's clearly a bug. But I don't believe it's what happened. The dumpxml above says check='full', which means the domain was successfully started and its CPU def was updated according to QEMU and thus the check attribute was changed to 'full'.
Noted.
The vme and arat features are added because libvirt's and QEMU's vision of the particual CPU model differs. Libvirt specifies it without vme and arat while QEMU has them both included in the CPU model. Thus when libvirt asks for that CPU model, QEMU enables these extra features too and libvirt adds them to the domain XML so that it can make sure they don't disappear when the domain is migrated or save/restored.
Thanks for that explanation.
So "somehow" QEMU added the CPU features 'vme' and 'arat' by itself, now you have to specify them in libvirt. So the admin ended up with a `sed` one-liner that updates the guest XML with the missing features:
sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear reason.
It was necessary mangling (the same as running `virt-xml`), as without adding the 'vme' and 'arat' features to their libvirt CPU definition, their guests wouldn't start anymore.
It would be nice to finally see what the admin wanted to achieve, what steps they did, and what result they saw.
I just double-confirmed with the admin, this was what happened: - They were running QEMU 2.6.5, VMs were starting fine. - Once they upgraded QEMU to 2.6.9, "suddenly" none of their VMs were starting, throwing the error about mismatch of guest CPU defintion (and the missing 'extra features'). - It was fixed once they explicitly added the two features ('vme' and 'arat') to the CPU guest XML.
Versions: libvirt 3.2 and QEMU 2.9
It's certainly possible there was a bug in 3.2 which got fixed since then (there were several of them), but it's impossible to guess without seeing what they're doing.
Noted above. -- /kashyap
On Wed, Jan 10, 2018 at 11:03:20 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote: ...
So "somehow" QEMU added the CPU features 'vme' and 'arat' by itself, now you have to specify them in libvirt. So the admin ended up with a `sed` one-liner that updates the guest XML with the missing features:
sed -i -e "s-</cpu>-<feature policy='require' name='vme'/></cpu>-" sed -i -e "s-</cpu>-<feature policy='require' name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear reason.
It was necessary mangling (the same as running `virt-xml`), as without adding the 'vme' and 'arat' features to their libvirt CPU definition, their guests wouldn't start anymore.
It would be nice to finally see what the admin wanted to achieve, what steps they did, and what result they saw.
I just double-confirmed with the admin, this was what happened:
- They were running QEMU 2.6.5, VMs were starting fine.
- Once they upgraded QEMU to 2.6.9, "suddenly" none of their VMs were starting, throwing the error about mismatch of guest CPU defintion (and the missing 'extra features').
Did they stored the XMLs dumped while the original domain was running and used them to start the domains on newer QEMU? The XML definition stored in libvirt should still have the original check='partial' and the domain should start fine after upgrading QEMU. But even with check='full' the domain should start fine since QEMU shouldn't be adding new features unless they also changed machine type in the XML. If the check='full' is stored in the inactive XML in /etc/libvirt/qemu, then this is something we need to look at. Jirka
On Wed, Jan 10, 2018 at 11:17:21AM +0100, Jiri Denemark wrote:
On Wed, Jan 10, 2018 at 11:03:20 +0100, Kashyap Chamarthy wrote:
[...]
I just double-confirmed with the admin, this was what happened:
- They were running QEMU 2.6.5, VMs were starting fine.
- Once they upgraded QEMU to 2.6.9, "suddenly" none of their VMs were starting, throwing the error about mismatch of guest CPU defintion (and the missing 'extra features').
Did they stored the XMLs dumped while the original domain was running and used them to start the domains on newer QEMU?
No, they didn't do something like that. (But I asked them the following.) [kashyap] So you didn't do any dumping of guest XMLs and using them to start the guests on updated QEMU? [admin] No. [admin] These XML files were generated by libvirt. [kashyap] So you simply updated QEMU, the guest was offline, and starting the guest failed w/ the missing CPU features. Yes? [admin] At least, for this guest that I was talking about I can't guarantee (I'm 99% sure we didn't migrate it; but can't say 100%). I can test it with another one if you want
The XML definition stored in libvirt should still have the original check='partial' and the domain should start fine after upgrading QEMU. But even with check='full' the domain should start fine since QEMU shouldn't be adding new features unless they also changed machine type in the XML.
If the check='full' is stored in the inactive XML in /etc/libvirt/qemu, then this is something we need to look at.
The admin reports that: "So, I am pretty sure that it was stored in the inactivce XML at some point, because of the fact that I had the error" I'll mention them to keep an eye on it. But he could confirm that with libvirt 3.2.0 and QEMU 2.9.0, creating new guests with `virt-install` does give him, correctly, check='partial'. -- /kashyap
participants (3)
-
Jiri Denemark -
Kashyap Chamarthy -
Pavel Hrdina