8:32 a.m.
The 'check' attribute is referring to this:
<cpu mode='custom' match='exact' check='full'>
Upstream documentation says, it is:
used to request a specific way of checking whether the virtual CPU
matches the specification. It is usually safe to omit this attribute
when starting a domain and stick with the default value.
But doesn't tell *what* the default value is. It is check='partial'.
Mention it so.
Thanks: Jiri Denemark
---
docs/formatdomain.html.in | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in
index d272cc1ba..8985da836 100644
--- a/docs/formatdomain.html.in
+++ b/docs/formatdomain.html.in
@@ -1287,10 +1287,11 @@
attribute can be used to request a specific way of checking whether
the virtual CPU matches the specification. It is usually safe to omit
this attribute when starting a domain and stick with the default
- value. Once the domain starts, libvirt will automatically change the
- <code>check</code> attribute to the best supported value to ensure
the
- virtual CPU does not change when the domain is migrated to another
- host. The following values can be used:
+ value (<code>partial>/code>). Once the domain starts, libvirt will
+ automatically change the <code>check</code> attribute to the best
+ supported value to ensure the virtual CPU does not change when the
+ domain is migrated to another host. The following values can be
+ used:
<dl>
<dt><code>none</code></dt>
@@ -1303,7 +1304,7 @@
<dt><code>partial</code></dt>
<dd>Libvirt will check the guest CPU specification before starting
a domain, but the rest is left on the hypervisor. It can still
- provide a different virtual CPU.</dd>
+ provide a different virtual CPU. (This is the default.)</dd>
<dt><code>full</code></dt>
<dd>The virtual CPU created by the hypervisor will be checked
--
2.13.6
9:37 a.m.
On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
The 'check' attribute is referring to this:
<cpu mode='custom' match='exact' check='full'>
Upstream documentation says, it is:
used to request a specific way of checking whether the virtual CPU
matches the specification. It is usually safe to omit this attribute
when starting a domain and stick with the default value.
But doesn't tell *what* the default value is. It is check='partial'.
Mention it so.
Thanks: Jiri Denemark
---
docs/formatdomain.html.in | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
NACK
As I said on IRC, the default differs with guest architecture ('partial'
for x86_64, 'none' for all others). Moreover this only applies to QEMU
driver. Oh and it even changes based on the mode attribute to make not
make it easy. But this is really an implementation detail. Libvirt will
just use a suitable default to stay compatible with older releases.
Jirka
12:44 p.m.
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
[...]
> But doesn't tell *what* the default value is. It is
check='partial'.
> Mention it so.
[...]
NACK
As I said on IRC, the default differs with guest architecture ('partial'
for x86_64, 'none' for all others).
Yep, I missed to distinguish that detail.
Moreover this only applies to QEMU driver. Oh and it even changes
based on the mode attribute to make not make it easy. But this is
really an implementation detail. Libvirt will just use a suitable
default to stay compatible with older releases.
That's all good. If you think nothing needs to be clarified here,
that's OK. (Some admins who operate a lot of VMs with libvirt will
disagree with you, though.)
And as we saw on IRC, `virt-install` muddies the water in this case by
gratuitously adding a CPU parameter with check='full' (even if a user
didn't explicitly specify it) when you simply import an image and look
at the run-time guest XML:
$> virt-install --name vm1 --ram 2048 \
--disk path=./vm1.qcow2,format=qcow2 --nographics \
--import --os-variant fedora27
$> virsh dumpxml vm1 | grep check
<cpu mode='custom' match='exact' check='full'>
(You might, fairly, argue here that: "Well, that's a bug in
`virt-install`, go complain there.")
It all comes across as messy to an unsuspecting admin. (If a competent
admin finds it confusing, good luck to new admins who want to wrap their
head around this.)
- - -
Background for others reading: The admin who reported this was confused
when he was creating guests with `virt-install`, which adds check='full'
(as noted earlier), and the guest throws:
error: Failed to start domain foo.org
error: operation failed: guest CPU doesn't match specification: extra features:
vme,arat
So "somehow" QEMU added the CPU features 'vme' and 'arat' by
itself, now
you have to specify them in libvirt. So the admin ended up with a `sed`
one-liner that updates the guest XML with the missing features:
sed -i -e "s-</cpu>-<feature policy='require'
name='vme'/></cpu>-"
sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
Versions: libvirt 3.2 and QEMU 2.9
1:28 p.m.
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
> On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
[...]
> > But doesn't tell *what* the default value is. It is
check='partial'.
> > Mention it so.
[...]
> NACK
>
> As I said on IRC, the default differs with guest architecture ('partial'
> for x86_64, 'none' for all others).
Yep, I missed to distinguish that detail.
> Moreover this only applies to QEMU driver. Oh and it even changes
> based on the mode attribute to make not make it easy. But this is
> really an implementation detail. Libvirt will just use a suitable
> default to stay compatible with older releases.
That's all good. If you think nothing needs to be clarified here,
that's OK. (Some admins who operate a lot of VMs with libvirt will
disagree with you, though.)
I'm not saying there's nothing to be clarified. Just that explicitly
specifying the default value will not help at all since it's complicated
and confusing by itself especially because of backward compatibility.
And as we saw on IRC, `virt-install` muddies the water in this case
by
gratuitously adding a CPU parameter with check='full' (even if a user
didn't explicitly specify it) when you simply import an image and look
at the run-time guest XML:
$> virt-install --name vm1 --ram 2048 \
--disk path=./vm1.qcow2,format=qcow2 --nographics \
--import --os-variant fedora27
$> virsh dumpxml vm1 | grep check
<cpu mode='custom' match='exact' check='full'>
(You might, fairly, argue here that: "Well, that's a bug in
`virt-install`, go complain there.")
I guess it's just the default behavior of virt-install and it
automatically passes the host CPU model to domain XML (without check
attribute, relying on the default) when creating new domains. After all,
any CPU model is going to be better than the default qemu64.
It all comes across as messy to an unsuspecting admin. (If a
competent
admin finds it confusing, good luck to new admins who want to wrap their
head around this.)
- - -
Background for others reading: The admin who reported this was confused
when he was creating guests with `virt-install`, which adds check='full'
(as noted earlier), and the guest throws:
error: Failed to start domain foo.org
error: operation failed: guest CPU doesn't match specification: extra features:
vme,arat
If this really happens when creating a domain with virt-install, then
it's clearly a bug. But I don't believe it's what happened. The dumpxml
above says check='full', which means the domain was successfully started
and its CPU def was updated according to QEMU and thus the check
attribute was changed to 'full'.
The vme and arat features are added because libvirt's and QEMU's vision
of the particual CPU model differs. Libvirt specifies it without vme and
arat while QEMU has them both included in the CPU model. Thus when
libvirt asks for that CPU model, QEMU enables these extra features too
and libvirt adds them to the domain XML so that it can make sure they
don't disappear when the domain is migrated or save/restored.
So "somehow" QEMU added the CPU features 'vme' and
'arat' by itself, now
you have to specify them in libvirt. So the admin ended up with a `sed`
one-liner that updates the guest XML with the missing features:
sed -i -e "s-</cpu>-<feature policy='require'
name='vme'/></cpu>-"
sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear
reason. It would be nice to finally see what the admin wanted to
achieve, what steps they did, and what result they saw.
Versions: libvirt 3.2 and QEMU 2.9
It's certainly possible there was a bug in 3.2 which got fixed since
then (there were several of them), but it's impossible to guess without
seeing what they're doing.
Jirka
2:51 a.m.
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
> On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
> > On Tue, Jan 09, 2018 at 15:32:54 +0100, Kashyap Chamarthy wrote:
>
> [...]
>
> > > But doesn't tell *what* the default value is. It is
check='partial'.
> > > Mention it so.
>
> [...]
>
> > NACK
> >
> > As I said on IRC, the default differs with guest architecture
('partial'
> > for x86_64, 'none' for all others).
>
> Yep, I missed to distinguish that detail.
>
> > Moreover this only applies to QEMU driver. Oh and it even changes
> > based on the mode attribute to make not make it easy. But this is
> > really an implementation detail. Libvirt will just use a suitable
> > default to stay compatible with older releases.
>
> That's all good. If you think nothing needs to be clarified here,
> that's OK. (Some admins who operate a lot of VMs with libvirt will
> disagree with you, though.)
I'm not saying there's nothing to be clarified. Just that explicitly
specifying the default value will not help at all since it's complicated
and confusing by itself especially because of backward compatibility.
> And as we saw on IRC, `virt-install` muddies the water in this case by
> gratuitously adding a CPU parameter with check='full' (even if a user
> didn't explicitly specify it) when you simply import an image and look
> at the run-time guest XML:
>
> $> virt-install --name vm1 --ram 2048 \
> --disk path=./vm1.qcow2,format=qcow2 --nographics \
> --import --os-variant fedora27
>
> $> virsh dumpxml vm1 | grep check
> <cpu mode='custom' match='exact' check='full'>
>
> (You might, fairly, argue here that: "Well, that's a bug in
> `virt-install`, go complain there.")
I guess it's just the default behavior of virt-install and it
automatically passes the host CPU model to domain XML (without check
attribute, relying on the default) when creating new domains. After all,
any CPU model is going to be better than the default qemu64.
One additional note for that virt-install command, this is the cpu
part of XML definition created by virt-install:
<cpu mode="custom" match="exact">
<model>Broadwell</model>
</cpu>
The cpu model will be different based on the host HW, but
as you can see, there is no "check='full'.
You can verify it by appending "--dry-run --print-xml" to your
virt-install command which will do nothing and only print the
XML definition.
> It all comes across as messy to an unsuspecting admin. (If a
competent
> admin finds it confusing, good luck to new admins who want to wrap their
> head around this.)
>
> - - -
>
> Background for others reading: The admin who reported this was confused
> when he was creating guests with `virt-install`, which adds check='full'
> (as noted earlier), and the guest throws:
>
> error: Failed to start domain foo.org
> error: operation failed: guest CPU doesn't match specification: extra
features: vme,arat
If this really happens when creating a domain with virt-install, then
it's clearly a bug. But I don't believe it's what happened. The dumpxml
above says check='full', which means the domain was successfully started
and its CPU def was updated according to QEMU and thus the check
attribute was changed to 'full'.
The vme and arat features are added because libvirt's and QEMU's vision
of the particual CPU model differs. Libvirt specifies it without vme and
arat while QEMU has them both included in the CPU model. Thus when
libvirt asks for that CPU model, QEMU enables these extra features too
and libvirt adds them to the domain XML so that it can make sure they
don't disappear when the domain is migrated or save/restored.
> So "somehow" QEMU added the CPU features 'vme' and 'arat'
by itself, now
> you have to specify them in libvirt. So the admin ended up with a `sed`
> one-liner that updates the guest XML with the missing features:
>
> sed -i -e "s-</cpu>-<feature policy='require'
name='vme'/></cpu>-"
> sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear
reason. It would be nice to finally see what the admin wanted to
achieve, what steps they did, and what result they saw.
Agreed, this is a mess. Just a note, if you need to edit XML
and you would like to automate it, you can use virt-xml:
virt-xml $domain --edit --cpu require=vme,require=arat
Anyway, it shouldn't be needed to modify the XML in the first place.
Pavel
4:13 a.m.
On Wed, Jan 10, 2018 at 09:51:19AM +0100, Pavel Hrdina wrote:
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
> On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
> > On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
[...]
One additional note for that virt-install command, this is the cpu
part of XML definition created by virt-install:
<cpu mode="custom" match="exact">
<model>Broadwell</model>
</cpu>
The cpu model will be different based on the host HW, but
as you can see, there is no "check='full'.
Noted. The admin has noticed the difference between the offline and
run-time definitions
You can verify it by appending "--dry-run --print-xml" to
your
virt-install command which will do nothing and only print the
XML definition.
Yep, this too the admin is aware.
Aside: is it expected that the run-time and offline config is
intentionaly different for the 'check' attribute?
Run-time:
$> virsh start vm1
$> virsh dumpxml vm1 | grep check
<cpu mode='custom' match='exact' check='full'>
^^^^
Offline:
$> virsh destroy vm1
$> virsh dumpxml vm1 | grep check
<cpu mode='custom' match='exact' check='partial'>
^^^^^^^
(Tried with: libvirt-daemon-kvm-3.7.0-2 and qemu-system-x86-2.10.0-4.)
[...]
> > sed -i -e "s-</cpu>-<feature
policy='require' name='vme'/></cpu>-"
> > sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
>
> This is some strange mangling of the XML by the admin for unclear
> reason. It would be nice to finally see what the admin wanted to
> achieve, what steps they did, and what result they saw.
Agreed, this is a mess.
No, the XML updating is not the "mess" (that was required in his case).
The mess is how all this is subtly opaque to the admin.
Just a note, if you need to edit XML
and you would like to automate it, you can use virt-xml:
virt-xml $domain --edit --cpu require=vme,require=arat
Yes, I already pointed that to the admin when he showed the `sed`
variant.
Anyway, it shouldn't be needed to modify the XML in the first
place.
It _was_ needed in his case (where updating QEMU broke). See my other
reply to Jiri.
--
/kashyap
4:25 a.m.
On Wed, Jan 10, 2018 at 11:13:28 +0100, Kashyap Chamarthy wrote:
On Wed, Jan 10, 2018 at 09:51:19AM +0100, Pavel Hrdina wrote:
> On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
> > On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
> > > On Tue, Jan 09, 2018 at 04:37:10PM +0100, Jiri Denemark wrote:
[...]
> One additional note for that virt-install command, this is the cpu
> part of XML definition created by virt-install:
>
> <cpu mode="custom" match="exact">
> <model>Broadwell</model>
> </cpu>
>
> The cpu model will be different based on the host HW, but
> as you can see, there is no "check='full'.
Noted. The admin has noticed the difference between the offline and
run-time definitions
> You can verify it by appending "--dry-run --print-xml" to your
> virt-install command which will do nothing and only print the
> XML definition.
Yep, this too the admin is aware.
Aside: is it expected that the run-time and offline config is
intentionaly different for the 'check' attribute?
Yes.
Run-time:
$> virsh start vm1
$> virsh dumpxml vm1 | grep check
<cpu mode='custom' match='exact' check='full'>
^^^^
This will make sure that the CPU will not change at all (no features
added or removed) when the running domain is migrated, save/restored, or
snapshotted and reverted.
Offline:
$> virsh destroy vm1
$> virsh dumpxml vm1 | grep check
<cpu mode='custom' match='exact' check='partial'>
This is needed for backward compatibility (old libvirt allowed you to
ask for some features if they were present in host CPU even though
QEMU/KVM did not support them) and to mitigate the differences between
libvirt's and QEMU's versions of CPU models (which may even differ based
on machine type). Once the domain is started, libvirt notes all the
small differences in the XML and changes check to 'full'.
(Tried with: libvirt-daemon-kvm-3.7.0-2 and
qemu-system-x86-2.10.0-4.)
[...]
> > > sed -i -e "s-</cpu>-<feature policy='require'
name='vme'/></cpu>-"
> > > sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
> >
> > This is some strange mangling of the XML by the admin for unclear
> > reason. It would be nice to finally see what the admin wanted to
> > achieve, what steps they did, and what result they saw.
>
> Agreed, this is a mess.
No, the XML updating is not the "mess" (that was required in his case).
The mess is how all this is subtly opaque to the admin.
> Just a note, if you need to edit XML
> and you would like to automate it, you can use virt-xml:
>
> virt-xml $domain --edit --cpu require=vme,require=arat
Yes, I already pointed that to the admin when he showed the `sed`
variant.
> Anyway, it shouldn't be needed to modify the XML in the first place.
It _was_ needed in his case (where updating QEMU broke).
It shouldn't be needed in any case though. It's either a bug or
something wrong they did before.
Jirka
4:03 a.m.
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
[...]
I'm not saying there's nothing to be clarified. Just that
explicitly
specifying the default value will not help at all since it's complicated
and confusing by itself especially because of backward compatibility.
Yeah, that I agree with you. It was just a v1 :-) (It would've been
clearer if youd've said: "NACK — needs some fixing".
[...]
> $> virt-install --name vm1 --ram 2048 \
> --disk path=./vm1.qcow2,format=qcow2 --nographics \
> --import --os-variant fedora27
>
> $> virsh dumpxml vm1 | grep check
> <cpu mode='custom' match='exact' check='full'>
>
> (You might, fairly, argue here that: "Well, that's a bug in
> `virt-install`, go complain there.")
I guess it's just the default behavior of virt-install and it
automatically passes the host CPU model to domain XML (without check
attribute, relying on the default) when creating new domains. After all,
any CPU model is going to be better than the default qemu64.
Right.
> Background for others reading: The admin who reported this was
confused
> when he was creating guests with `virt-install`, which adds check='full'
> (as noted earlier), and the guest throws:
>
> error: Failed to start domain foo.org
> error: operation failed: guest CPU doesn't match specification: extra
features: vme,arat
If this really happens when creating a domain with virt-install,
Right, it doesn't happen on domain start (I worded it poorly). The
error occurred only after QEMU was updated (from 2.6.5 to 2.9).
then it's clearly a bug. But I don't believe it's what
happened. The
dumpxml above says check='full', which means the domain was
successfully started and its CPU def was updated according to QEMU and
thus the check attribute was changed to 'full'.
Noted.
The vme and arat features are added because libvirt's and
QEMU's vision
of the particual CPU model differs. Libvirt specifies it without vme and
arat while QEMU has them both included in the CPU model. Thus when
libvirt asks for that CPU model, QEMU enables these extra features too
and libvirt adds them to the domain XML so that it can make sure they
don't disappear when the domain is migrated or save/restored.
Thanks for that explanation.
> So "somehow" QEMU added the CPU features 'vme'
and 'arat' by itself, now
> you have to specify them in libvirt. So the admin ended up with a `sed`
> one-liner that updates the guest XML with the missing features:
>
> sed -i -e "s-</cpu>-<feature policy='require'
name='vme'/></cpu>-"
> sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
This is some strange mangling of the XML by the admin for unclear
reason.
It was necessary mangling (the same as running `virt-xml`), as without
adding the 'vme' and 'arat' features to their libvirt CPU definition,
their guests wouldn't start anymore.
It would be nice to finally see what the admin wanted to
achieve, what steps they did, and what result they saw.
I just double-confirmed with the admin, this was what happened:
- They were running QEMU 2.6.5, VMs were starting fine.
- Once they upgraded QEMU to 2.6.9, "suddenly" none of their VMs were
starting, throwing the error about mismatch of guest CPU defintion
(and the missing 'extra features').
- It was fixed once they explicitly added the two features ('vme' and
'arat') to the CPU guest XML.
> Versions: libvirt 3.2 and QEMU 2.9
It's certainly possible there was a bug in 3.2 which got fixed since
then (there were several of them), but it's impossible to guess without
seeing what they're doing.
Noted above.
--
/kashyap
4:17 a.m.
On Wed, Jan 10, 2018 at 11:03:20 +0100, Kashyap Chamarthy wrote:
On Tue, Jan 09, 2018 at 08:28:20PM +0100, Jiri Denemark wrote:
> On Tue, Jan 09, 2018 at 19:44:18 +0100, Kashyap Chamarthy wrote:
...
> > So "somehow" QEMU added the CPU features
'vme' and 'arat' by itself, now
> > you have to specify them in libvirt. So the admin ended up with a `sed`
> > one-liner that updates the guest XML with the missing features:
> >
> > sed -i -e "s-</cpu>-<feature policy='require'
name='vme'/></cpu>-"
> > sed -i -e "s-</cpu>-<feature policy='require'
name='arat'/></cpu>-"
>
> This is some strange mangling of the XML by the admin for unclear
> reason.
It was necessary mangling (the same as running `virt-xml`), as without
adding the 'vme' and 'arat' features to their libvirt CPU definition,
their guests wouldn't start anymore.
> It would be nice to finally see what the admin wanted to
> achieve, what steps they did, and what result they saw.
I just double-confirmed with the admin, this was what happened:
- They were running QEMU 2.6.5, VMs were starting fine.
- Once they upgraded QEMU to 2.6.9, "suddenly" none of their VMs were
starting, throwing the error about mismatch of guest CPU defintion
(and the missing 'extra features').
Did they stored the XMLs dumped while the original domain was running
and used them to start the domains on newer QEMU? The XML definition
stored in libvirt should still have the original check='partial' and the
domain should start fine after upgrading QEMU. But even with
check='full' the domain should start fine since QEMU shouldn't be adding
new features unless they also changed machine type in the XML.
If the check='full' is stored in the inactive XML in /etc/libvirt/qemu,
then this is something we need to look at.
Jirka
8:54 a.m.
On Wed, Jan 10, 2018 at 11:17:21AM +0100, Jiri Denemark wrote:
On Wed, Jan 10, 2018 at 11:03:20 +0100, Kashyap Chamarthy wrote:
[...]
> I just double-confirmed with the admin, this was what happened:
>
> - They were running QEMU 2.6.5, VMs were starting fine.
>
> - Once they upgraded QEMU to 2.6.9, "suddenly" none of their VMs were
> starting, throwing the error about mismatch of guest CPU defintion
> (and the missing 'extra features').
Did they stored the XMLs dumped while the original domain was running
and used them to start the domains on newer QEMU?
No, they didn't do something like that. (But I asked them the
following.)
[kashyap] So you didn't do any dumping of guest XMLs and using them to
start the guests on updated QEMU?
[admin] No.
[admin] These XML files were generated by libvirt.
[kashyap] So you simply updated QEMU, the guest was offline, and
starting the guest failed w/ the missing CPU features. Yes?
[admin] At least, for this guest that I was talking about I can't
guarantee (I'm 99% sure we didn't migrate it; but can't
say 100%). I can test it with another one if you want
The XML definition stored in libvirt should still have the original
check='partial' and the domain should start fine after upgrading QEMU.
But even with check='full' the domain should start fine since QEMU
shouldn't be adding new features unless they also changed machine type
in the XML.
If the check='full' is stored in the inactive XML in
/etc/libvirt/qemu, then this is something we need to look at.
The admin reports that: "So, I am pretty sure that it was stored in the
inactivce XML at some point, because of the fact that I had the error"
I'll mention them to keep an eye on it.
But he could confirm that with libvirt 3.2.0 and QEMU 2.9.0, creating
new guests with `virt-install` does give him, correctly,
check='partial'.
--
/kashyap
2508
days inactive
2509
days old
9 comments
3 participants
participants (3)
-
Jiri Denemark -
Kashyap Chamarthy -
Pavel Hrdina