9:04 p.m.
Hi:
I update our RHEL9 system to RHEL 9.4, which brings libvirt 10.0.
I try to calculate the cpu baseline for our two-node cluster with
command "virsh domcapabilities" then "virsh hypervisor-cpu-baseline
--migratable". the result has many cpu features begin with "vmx".
the test cluster has cpu "intel E3-1280 V3" and "intel I3-9100F".
when I try live migrate vm, it failed and told me "guest CPU doesn't
match specification: missing features:
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr".
at another cluster with cpu " Intel i5-2520M" and "Intel
i7-9750H" the migration works fine for the calculated cpu result.
although there are still many "vmx" cpu features in the result.
if I delete all these vmx features, then migration works fine for
both cluster, like old days.
I wonder what's the benefit to expose these vmx features to guest
if I don't do any nested virtualization.
is it ok the drop all these vmx cpu features for a guest?
thanks a lot for help!
5:48 p.m.
Hi.
On Mon, May 20, 2024 at 21:04:59 +0800, d tbsky wrote:
Hi:
I update our RHEL9 system to RHEL 9.4, which brings libvirt 10.0.
I try to calculate the cpu baseline for our two-node cluster with
command "virsh domcapabilities" then "virsh hypervisor-cpu-baseline
--migratable". the result has many cpu features begin with "vmx".
the test cluster has cpu "intel E3-1280 V3" and "intel
I3-9100F".
when I try live migrate vm, it failed and told me "guest CPU doesn't
match specification: missing features:
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr".
Interesting, hypervisor-cpu-baseline is supposed to provide an
intersection of both CPU models. I wonder whether we have a bug
somewhere, could you share the host-model definition from virsh
domcapabilities from both hosts? The easiest way is running
$ virsh domcapabilities | xmllint --xpath
'//cpu/mode[@name="host-model"]' -
at another cluster with cpu " Intel i5-2520M" and
"Intel
i7-9750H" the migration works fine for the calculated cpu result.
although there are still many "vmx" cpu features in the result.
if I delete all these vmx features, then migration works fine for
both cluster, like old days.
I wonder what's the benefit to expose these vmx features to guest
if I don't do any nested virtualization.
Well, they were always exposed by QEMU, libvirt just added support for
them and thus they are visible in the XMLs as well.
is it ok the drop all these vmx cpu features for a guest?
thanks a lot for help!
Sure, unless you specifically need some of them, you can drop them. They
will be added to the XML anyway during VM startup.
Jirka
3:59 p.m.
Hi:
Jiri Denemark <jdenemar(a)redhat.com>
Interesting, hypervisor-cpu-baseline is supposed to provide an
intersection of both CPU models. I wonder whether we have a bug
somewhere, could you share the host-model definition from virsh
domcapabilities from both hosts? The easiest way is running
$ virsh domcapabilities | xmllint --xpath
'//cpu/mode[@name="host-model"]' -
the result are below:
for cpu i3-9100F:
<mode name="host-model" supported="yes">
<model fallback="forbid">Skylake-Client-IBRS</model>
<vendor>Intel</vendor>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="vmx"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="clflushopt"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaves"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="invtsc"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="rsba"/>
<feature policy="require" name="pschange-mc-no"/>
<feature policy="require" name="vmx-ins-outs"/>
<feature policy="require" name="vmx-true-ctls"/>
<feature policy="require" name="vmx-store-lma"/>
<feature policy="require" name="vmx-activity-hlt"/>
<feature policy="require" name="vmx-activity-wait-sipi"/>
<feature policy="require"
name="vmx-vmwrite-vmexit-fields"/>
<feature policy="require" name="vmx-apicv-xapic"/>
<feature policy="require" name="vmx-ept"/>
<feature policy="require" name="vmx-desc-exit"/>
<feature policy="require" name="vmx-rdtscp-exit"/>
<feature policy="require" name="vmx-apicv-x2apic"/>
<feature policy="require" name="vmx-vpid"/>
<feature policy="require" name="vmx-wbinvd-exit"/>
<feature policy="require" name="vmx-unrestricted-guest"/>
<feature policy="require" name="vmx-rdrand-exit"/>
<feature policy="require" name="vmx-invpcid-exit"/>
<feature policy="require" name="vmx-vmfunc"/>
<feature policy="require" name="vmx-shadow-vmcs"/>
<feature policy="require" name="vmx-rdseed-exit"/>
<feature policy="require" name="vmx-pml"/>
<feature policy="require" name="vmx-xsaves"/>
<feature policy="require" name="vmx-invvpid"/>
<feature policy="require"
name="vmx-invvpid-single-addr"/>
<feature policy="require"
name="vmx-invvpid-all-context"/>
<feature policy="require" name="vmx-ept-execonly"/>
<feature policy="require" name="vmx-page-walk-4"/>
<feature policy="require" name="vmx-ept-2mb"/>
<feature policy="require" name="vmx-ept-1gb"/>
<feature policy="require" name="vmx-invept"/>
<feature policy="require" name="vmx-eptad"/>
<feature policy="require"
name="vmx-invept-single-context"/>
<feature policy="require" name="vmx-invept-all-context"/>
<feature policy="require" name="vmx-intr-exit"/>
<feature policy="require" name="vmx-nmi-exit"/>
<feature policy="require" name="vmx-vnmi"/>
<feature policy="require" name="vmx-preemption-timer"/>
<feature policy="require" name="vmx-vintr-pending"/>
<feature policy="require" name="vmx-tsc-offset"/>
<feature policy="require" name="vmx-hlt-exit"/>
<feature policy="require" name="vmx-invlpg-exit"/>
<feature policy="require" name="vmx-mwait-exit"/>
<feature policy="require" name="vmx-rdpmc-exit"/>
<feature policy="require" name="vmx-rdtsc-exit"/>
<feature policy="require" name="vmx-cr3-load-noexit"/>
<feature policy="require" name="vmx-cr3-store-noexit"/>
<feature policy="require" name="vmx-cr8-load-exit"/>
<feature policy="require" name="vmx-cr8-store-exit"/>
<feature policy="require" name="vmx-flexpriority"/>
<feature policy="require" name="vmx-vnmi-pending"/>
<feature policy="require" name="vmx-movdr-exit"/>
<feature policy="require" name="vmx-io-exit"/>
<feature policy="require" name="vmx-io-bitmap"/>
<feature policy="require" name="vmx-mtf"/>
<feature policy="require" name="vmx-msr-bitmap"/>
<feature policy="require" name="vmx-monitor-exit"/>
<feature policy="require" name="vmx-pause-exit"/>
<feature policy="require" name="vmx-secondary-ctls"/>
<feature policy="require"
name="vmx-exit-nosave-debugctl"/>
<feature policy="require"
name="vmx-exit-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-exit-ack-intr"/>
<feature policy="require" name="vmx-exit-save-pat"/>
<feature policy="require" name="vmx-exit-load-pat"/>
<feature policy="require" name="vmx-exit-save-efer"/>
<feature policy="require" name="vmx-exit-load-efer"/>
<feature policy="require"
name="vmx-exit-save-preemption-timer"/>
<feature policy="require" name="vmx-exit-clear-bndcfgs"/>
<feature policy="require"
name="vmx-entry-noload-debugctl"/>
<feature policy="require" name="vmx-entry-ia32e-mode"/>
<feature policy="require"
name="vmx-entry-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-entry-load-pat"/>
<feature policy="require" name="vmx-entry-load-efer"/>
<feature policy="require" name="vmx-entry-load-bndcfgs"/>
<feature policy="require" name="vmx-eptp-switching"/>
<feature policy="disable" name="hle"/>
<feature policy="disable" name="rtm"/>
</mode>
for cpu E3-1280 V3:
<mode name="host-model" supported="yes">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="vme"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="vmx"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="f16c"/>
<feature policy="require" name="rdrand"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="arat"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaveopt"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="abm"/>
<feature policy="require" name="invtsc"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="pschange-mc-no"/>
<feature policy="require" name="gds-no"/>
<feature policy="require" name="vmx-ins-outs"/>
<feature policy="require" name="vmx-true-ctls"/>
<feature policy="require" name="vmx-store-lma"/>
<feature policy="require" name="vmx-activity-hlt"/>
<feature policy="require" name="vmx-activity-wait-sipi"/>
<feature policy="require"
name="vmx-vmwrite-vmexit-fields"/>
<feature policy="require" name="vmx-apicv-xapic"/>
<feature policy="require" name="vmx-ept"/>
<feature policy="require" name="vmx-desc-exit"/>
<feature policy="require" name="vmx-rdtscp-exit"/>
<feature policy="require" name="vmx-apicv-x2apic"/>
<feature policy="require" name="vmx-vpid"/>
<feature policy="require" name="vmx-wbinvd-exit"/>
<feature policy="require" name="vmx-unrestricted-guest"/>
<feature policy="require" name="vmx-rdrand-exit"/>
<feature policy="require" name="vmx-invpcid-exit"/>
<feature policy="require" name="vmx-vmfunc"/>
<feature policy="require" name="vmx-shadow-vmcs"/>
<feature policy="require" name="vmx-pml"/>
<feature policy="require" name="vmx-invvpid"/>
<feature policy="require"
name="vmx-invvpid-single-addr"/>
<feature policy="require"
name="vmx-invvpid-all-context"/>
<feature policy="require" name="vmx-ept-execonly"/>
<feature policy="require" name="vmx-page-walk-4"/>
<feature policy="require" name="vmx-ept-2mb"/>
<feature policy="require" name="vmx-ept-1gb"/>
<feature policy="require" name="vmx-invept"/>
<feature policy="require" name="vmx-eptad"/>
<feature policy="require"
name="vmx-invept-single-context"/>
<feature policy="require" name="vmx-invept-all-context"/>
<feature policy="require" name="vmx-intr-exit"/>
<feature policy="require" name="vmx-nmi-exit"/>
<feature policy="require" name="vmx-vnmi"/>
<feature policy="require" name="vmx-preemption-timer"/>
<feature policy="require" name="vmx-vintr-pending"/>
<feature policy="require" name="vmx-tsc-offset"/>
<feature policy="require" name="vmx-hlt-exit"/>
<feature policy="require" name="vmx-invlpg-exit"/>
<feature policy="require" name="vmx-mwait-exit"/>
<feature policy="require" name="vmx-rdpmc-exit"/>
<feature policy="require" name="vmx-rdtsc-exit"/>
<feature policy="require" name="vmx-cr3-load-noexit"/>
<feature policy="require" name="vmx-cr3-store-noexit"/>
<feature policy="require" name="vmx-cr8-load-exit"/>
<feature policy="require" name="vmx-cr8-store-exit"/>
<feature policy="require" name="vmx-flexpriority"/>
<feature policy="require" name="vmx-vnmi-pending"/>
<feature policy="require" name="vmx-movdr-exit"/>
<feature policy="require" name="vmx-io-exit"/>
<feature policy="require" name="vmx-io-bitmap"/>
<feature policy="require" name="vmx-mtf"/>
<feature policy="require" name="vmx-msr-bitmap"/>
<feature policy="require" name="vmx-monitor-exit"/>
<feature policy="require" name="vmx-pause-exit"/>
<feature policy="require" name="vmx-secondary-ctls"/>
<feature policy="require"
name="vmx-exit-nosave-debugctl"/>
<feature policy="require"
name="vmx-exit-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-exit-ack-intr"/>
<feature policy="require" name="vmx-exit-save-pat"/>
<feature policy="require" name="vmx-exit-load-pat"/>
<feature policy="require" name="vmx-exit-save-efer"/>
<feature policy="require" name="vmx-exit-load-efer"/>
<feature policy="require"
name="vmx-exit-save-preemption-timer"/>
<feature policy="require"
name="vmx-entry-noload-debugctl"/>
<feature policy="require" name="vmx-entry-ia32e-mode"/>
<feature policy="require"
name="vmx-entry-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-entry-load-pat"/>
<feature policy="require" name="vmx-entry-load-efer"/>
<feature policy="require" name="vmx-eptp-switching"/>
</mode>
Well, they were always exposed by QEMU, libvirt just added support
for
them and thus they are visible in the XMLs as well.
indeed. I check the guest machine via "cat /proc/cpuinfo" and vmx
is still there even though I deleted these vmx lines.
is it ok the drop all these vmx cpu features for a guest?
thanks a lot for help!
Sure, unless you specifically need some of them, you can drop them. They
will be added to the XML anyway during VM startup.
indeed. i try to run "virsh dumpxml" for a running vm and I saw many
vmx lines.
so I got confused. is "virsh dumpxml" supposed to be the same as
"hypervisor-cpu-baseline" result?
PS: according to redhat
document(https://access.redhat.com/documentation/zh-tw/red_hat_enterprise...,
it use "hypervisor-cpu-baseline" without "--migratable". it will
bring
in "invtsc" cpu feature. I can never live migrate with the feature.
7:41 p.m.
On Wed, May 22, 2024 at 15:59:36 +0800, d tbsky wrote:
Hi:
Jiri Denemark <jdenemar(a)redhat.com>
> Interesting, hypervisor-cpu-baseline is supposed to provide an
> intersection of both CPU models. I wonder whether we have a bug
> somewhere, could you share the host-model definition from virsh
> domcapabilities from both hosts? The easiest way is running
>
> $ virsh domcapabilities | xmllint --xpath
'//cpu/mode[@name="host-model"]' -
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr
the result are below:
for cpu i3-9100F:
<mode name="host-model" supported="yes">
<model fallback="forbid">Skylake-Client-IBRS</model>
<vendor>Intel</vendor>
...
</mode>
for cpu E3-1280 V3:
<mode name="host-model" supported="yes">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
...
</mode>
Thanks, neither CPU model contains any of the features migration is
later complaining about (vmx-apicv-register, vmx-apicv-vid,
vmx-posted-intr). But they are all included in the Haswell-noTSX-IBRS
model while missing in Skylake-Client-IBRS. So I guess if you use the
baseline CPU model and start a domain using it on E3-1280 V3 the
features might be enabled by QEMU (because the baseline would not
explicitly disable them) and thus migrating to i3-9100F would result in
such error. Technically the error is correct as the guest could expect
the features to be present and try to use them after migration, which
would likely cause the guest to crash. On the other hand the computed
baseline CPU model would be wrong if my theory is correct.
So could you also provide
(1) the output of "virsh hypervisor-cpu-baseline" and
(2) the <cpu> element from the domain XML before it is started
(3) and once it is running on the host from which migration fails?
An easy way to extract the <cpu> element from domain XML is:
virsh dumpxml nest | xmllint --xpath '/domain/cpu' -
> is it ok the drop all these vmx cpu features for a guest?
> thanks a lot for help!
>
> Sure, unless you specifically need some of them, you can drop them. They
> will be added to the XML anyway during VM startup.
That said, it may not be safe because of the reason I explained above,
but as long as it works, it can be used as a workaround.
indeed. i try to run "virsh dumpxml" for a running vm and
I saw many
vmx lines.
so I got confused. is "virsh dumpxml" supposed to be the same as
"hypervisor-cpu-baseline" result?
Due to compatibility reasons libvirt's definition of specific CPU models
may differ from the definition QEMU uses so once a domain gets started
we query QEMU for the features that were actually enabled (as the set
may differ from what we asked for) and updates the domain XML
accordingly. Ideally when using a CPU definition computed by
hypervisor-cpu-baseline there should be no features added in domain XML
(virsh dumpxml) once the domain starts. In you case (as long as my
theory is correct) features that we have in our definition of
Haswell-noTSX-IBRS are added once domain starts and those features are
not available on the second host, which should never happen as such
features should have been explicitly disabled in the baseline CPU.
PS: according to redhat
document(https://access.redhat.com/documentation/zh-tw/red_hat_enterprise...,
it use "hypervisor-cpu-baseline" without "--migratable". it will
bring
in "invtsc" cpu feature. I can never live migrate with the feature.
Yes, without --migratable some features may be included in the result
that would block migration. They don't have to though, it depends the
common set of features among the hosts for which baseline is computed.
And invtsc is not the only non-migratable feature. In other words, using
--migratable for hypervisor-cpu-baseline is usually a good idea.
Jirka
1:20 a.m.
Hi:
Jiri Denemark <jdenemar(a)redhat.com>
So could you also provide
(1) the output of "virsh hypervisor-cpu-baseline" and
(2) the <cpu> element from the domain XML before it is started
(3) and once it is running on the host from which migration fails?
sorry when I checked the vm with your theory(qemu will enable vmx
automatically) in my last post, I wrongly checked a vm which is still
using RHEL 9.3 config. RHEL 9.3 config has "vmx" feature but not other
"vmx-blabla." sub features. When the vm config has "vmx" feature,
"virsh dumpxml" a running vm will bring all these "vmx-blabla" sub
feature lines at RHEL 9.4.
but if I delete the "vmx" feature from the vm config, then qemu
won't enable any "vmx" related feature. "vmx" doesn't exist
in guest
vm or "virsh dumpxml". and migration works as expected. xml are the
same when running at E3-1280-V3 and I3-9100F. the only difference
between shutdown/running xml is 'check="partial' and
'check="full"'.
virsh dumpxml my-vm | xmllint --xpath '/domain/cpu' -
<cpu mode="custom" match="exact" check="full">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" dies="1" clusters="1"
cores="2" threads="1"/>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="vme"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="f16c"/>
<feature policy="require" name="rdrand"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="arat"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaveopt"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="abm"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="pschange-mc-no"/>
</cpu>
Next test I add "vmx" feature to vm config(so it will look lik RHEL
9.3 config). the xml are the same at both cpu. live migration failed
at both direction and error message are the same: "guest CPU doesn't
match specification: missing features:
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr". we can see these
features were disabled at running xml.
shutdown xml below:
<cpu mode="custom" match="exact" check="partial">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" dies="1" clusters="1"
cores="1" threads="1"/>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="vmx"/>
<feature policy="require" name="vme"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="f16c"/>
<feature policy="require" name="rdrand"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="arat"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaveopt"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="abm"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="pschange-mc-no"/>
</cpu>
running xml below:
<cpu mode="custom" match="exact" check="full">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" dies="1" clusters="1"
cores="1" threads="1"/>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="vmx"/>
<feature policy="require" name="vme"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="f16c"/>
<feature policy="require" name="rdrand"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="arat"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaveopt"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="abm"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="pschange-mc-no"/>
<feature policy="require" name="vmx-ins-outs"/>
<feature policy="require" name="vmx-true-ctls"/>
<feature policy="require" name="vmx-store-lma"/>
<feature policy="require" name="vmx-activity-hlt"/>
<feature policy="require"
name="vmx-vmwrite-vmexit-fields"/>
<feature policy="require" name="vmx-apicv-xapic"/>
<feature policy="require" name="vmx-ept"/>
<feature policy="require" name="vmx-desc-exit"/>
<feature policy="require" name="vmx-rdtscp-exit"/>
<feature policy="require" name="vmx-apicv-x2apic"/>
<feature policy="require" name="vmx-vpid"/>
<feature policy="require" name="vmx-wbinvd-exit"/>
<feature policy="require" name="vmx-unrestricted-guest"/>
<feature policy="disable" name="vmx-apicv-register"/>
<feature policy="disable" name="vmx-apicv-vid"/>
<feature policy="require" name="vmx-rdrand-exit"/>
<feature policy="require" name="vmx-invpcid-exit"/>
<feature policy="require" name="vmx-vmfunc"/>
<feature policy="require" name="vmx-shadow-vmcs"/>
<feature policy="require" name="vmx-invvpid"/>
<feature policy="require" name="vmx-invvpid-single-addr"/>
<feature policy="require" name="vmx-invvpid-all-context"/>
<feature policy="require" name="vmx-ept-execonly"/>
<feature policy="require" name="vmx-page-walk-4"/>
<feature policy="require" name="vmx-ept-2mb"/>
<feature policy="require" name="vmx-ept-1gb"/>
<feature policy="require" name="vmx-invept"/>
<feature policy="require" name="vmx-eptad"/>
<feature policy="require"
name="vmx-invept-single-context"/>
<feature policy="require" name="vmx-invept-all-context"/>
<feature policy="require" name="vmx-intr-exit"/>
<feature policy="require" name="vmx-nmi-exit"/>
<feature policy="require" name="vmx-vnmi"/>
<feature policy="require" name="vmx-preemption-timer"/>
<feature policy="disable" name="vmx-posted-intr"/>
<feature policy="require" name="vmx-vintr-pending"/>
<feature policy="require" name="vmx-tsc-offset"/>
<feature policy="require" name="vmx-hlt-exit"/>
<feature policy="require" name="vmx-invlpg-exit"/>
<feature policy="require" name="vmx-mwait-exit"/>
<feature policy="require" name="vmx-rdpmc-exit"/>
<feature policy="require" name="vmx-rdtsc-exit"/>
<feature policy="require" name="vmx-cr3-load-noexit"/>
<feature policy="require" name="vmx-cr3-store-noexit"/>
<feature policy="require" name="vmx-cr8-load-exit"/>
<feature policy="require" name="vmx-cr8-store-exit"/>
<feature policy="require" name="vmx-flexpriority"/>
<feature policy="require" name="vmx-vnmi-pending"/>
<feature policy="require" name="vmx-movdr-exit"/>
<feature policy="require" name="vmx-io-exit"/>
<feature policy="require" name="vmx-io-bitmap"/>
<feature policy="require" name="vmx-mtf"/>
<feature policy="require" name="vmx-msr-bitmap"/>
<feature policy="require" name="vmx-monitor-exit"/>
<feature policy="require" name="vmx-pause-exit"/>
<feature policy="require" name="vmx-secondary-ctls"/>
<feature policy="require" name="vmx-exit-nosave-debugctl"/>
<feature policy="require" name="vmx-exit-ack-intr"/>
<feature policy="require" name="vmx-exit-save-pat"/>
<feature policy="require" name="vmx-exit-load-pat"/>
<feature policy="require" name="vmx-exit-save-efer"/>
<feature policy="require" name="vmx-exit-load-efer"/>
<feature policy="require"
name="vmx-exit-save-preemption-timer"/>
<feature policy="require"
name="vmx-entry-noload-debugctl"/>
<feature policy="require" name="vmx-entry-ia32e-mode"/>
<feature policy="require" name="vmx-entry-load-pat"/>
<feature policy="require" name="vmx-entry-load-efer"/>
<feature policy="require" name="vmx-eptp-switching"/>
</cpu>
the final test I try to use "hypervisor-cpu-baseline --migratable "
result. again the xml are the same at both cpu. the difference between
shutdown xml/running xml is the last three lines. failed cpu features
were disabled at running xml automatically
.
shutdown xml below:
<cpu mode="custom" match="exact" check="partial">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" dies="1" clusters="1"
cores="1" threads="1"/>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="vme"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="vmx"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="f16c"/>
<feature policy="require" name="rdrand"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="arat"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaveopt"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="abm"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="pschange-mc-no"/>
<feature policy="require" name="vmx-ins-outs"/>
<feature policy="require" name="vmx-true-ctls"/>
<feature policy="require" name="vmx-store-lma"/>
<feature policy="require" name="vmx-activity-hlt"/>
<feature policy="require" name="vmx-activity-wait-sipi"/>
<feature policy="require"
name="vmx-vmwrite-vmexit-fields"/>
<feature policy="require" name="vmx-apicv-xapic"/>
<feature policy="require" name="vmx-ept"/>
<feature policy="require" name="vmx-desc-exit"/>
<feature policy="require" name="vmx-rdtscp-exit"/>
<feature policy="require" name="vmx-apicv-x2apic"/>
<feature policy="require" name="vmx-vpid"/>
<feature policy="require" name="vmx-wbinvd-exit"/>
<feature policy="require" name="vmx-unrestricted-guest"/>
<feature policy="require" name="vmx-rdrand-exit"/>
<feature policy="require" name="vmx-invpcid-exit"/>
<feature policy="require" name="vmx-vmfunc"/>
<feature policy="require" name="vmx-shadow-vmcs"/>
<feature policy="require" name="vmx-pml"/>
<feature policy="require" name="vmx-invvpid"/>
<feature policy="require" name="vmx-invvpid-single-addr"/>
<feature policy="require" name="vmx-invvpid-all-context"/>
<feature policy="require" name="vmx-ept-execonly"/>
<feature policy="require" name="vmx-page-walk-4"/>
<feature policy="require" name="vmx-ept-2mb"/>
<feature policy="require" name="vmx-ept-1gb"/>
<feature policy="require" name="vmx-invept"/>
<feature policy="require" name="vmx-eptad"/>
<feature policy="require"
name="vmx-invept-single-context"/>
<feature policy="require" name="vmx-invept-all-context"/>
<feature policy="require" name="vmx-intr-exit"/>
<feature policy="require" name="vmx-nmi-exit"/>
<feature policy="require" name="vmx-vnmi"/>
<feature policy="require" name="vmx-preemption-timer"/>
<feature policy="require" name="vmx-vintr-pending"/>
<feature policy="require" name="vmx-tsc-offset"/>
<feature policy="require" name="vmx-hlt-exit"/>
<feature policy="require" name="vmx-invlpg-exit"/>
<feature policy="require" name="vmx-mwait-exit"/>
<feature policy="require" name="vmx-rdpmc-exit"/>
<feature policy="require" name="vmx-rdtsc-exit"/>
<feature policy="require" name="vmx-cr3-load-noexit"/>
<feature policy="require" name="vmx-cr3-store-noexit"/>
<feature policy="require" name="vmx-cr8-load-exit"/>
<feature policy="require" name="vmx-cr8-store-exit"/>
<feature policy="require" name="vmx-flexpriority"/>
<feature policy="require" name="vmx-vnmi-pending"/>
<feature policy="require" name="vmx-movdr-exit"/>
<feature policy="require" name="vmx-io-exit"/>
<feature policy="require" name="vmx-io-bitmap"/>
<feature policy="require" name="vmx-mtf"/>
<feature policy="require" name="vmx-msr-bitmap"/>
<feature policy="require" name="vmx-monitor-exit"/>
<feature policy="require" name="vmx-pause-exit"/>
<feature policy="require" name="vmx-secondary-ctls"/>
<feature policy="require" name="vmx-exit-nosave-debugctl"/>
<feature policy="require"
name="vmx-exit-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-exit-ack-intr"/>
<feature policy="require" name="vmx-exit-save-pat"/>
<feature policy="require" name="vmx-exit-load-pat"/>
<feature policy="require" name="vmx-exit-save-efer"/>
<feature policy="require" name="vmx-exit-load-efer"/>
<feature policy="require"
name="vmx-exit-save-preemption-timer"/>
<feature policy="require"
name="vmx-entry-noload-debugctl"/>
<feature policy="require" name="vmx-entry-ia32e-mode"/>
<feature policy="require"
name="vmx-entry-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-entry-load-pat"/>
<feature policy="require" name="vmx-entry-load-efer"/>
<feature policy="require" name="vmx-eptp-switching"/>
</cpu>
running xml below:
<cpu mode="custom" match="exact" check="full">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" dies="1" clusters="1"
cores="1" threads="1"/>
<maxphysaddr mode="passthrough" limit="39"/>
<feature policy="require" name="vme"/>
<feature policy="require" name="ss"/>
<feature policy="require" name="vmx"/>
<feature policy="require" name="pdcm"/>
<feature policy="require" name="f16c"/>
<feature policy="require" name="rdrand"/>
<feature policy="require" name="hypervisor"/>
<feature policy="require" name="arat"/>
<feature policy="require" name="tsc_adjust"/>
<feature policy="require" name="umip"/>
<feature policy="require" name="md-clear"/>
<feature policy="require" name="stibp"/>
<feature policy="require" name="flush-l1d"/>
<feature policy="require" name="arch-capabilities"/>
<feature policy="require" name="ssbd"/>
<feature policy="require" name="xsaveopt"/>
<feature policy="require" name="pdpe1gb"/>
<feature policy="require" name="abm"/>
<feature policy="require" name="ibpb"/>
<feature policy="require" name="ibrs"/>
<feature policy="require" name="amd-stibp"/>
<feature policy="require" name="amd-ssbd"/>
<feature policy="require" name="pschange-mc-no"/>
<feature policy="require" name="vmx-ins-outs"/>
<feature policy="require" name="vmx-true-ctls"/>
<feature policy="require" name="vmx-store-lma"/>
<feature policy="require" name="vmx-activity-hlt"/>
<feature policy="require" name="vmx-activity-wait-sipi"/>
<feature policy="require"
name="vmx-vmwrite-vmexit-fields"/>
<feature policy="require" name="vmx-apicv-xapic"/>
<feature policy="require" name="vmx-ept"/>
<feature policy="require" name="vmx-desc-exit"/>
<feature policy="require" name="vmx-rdtscp-exit"/>
<feature policy="require" name="vmx-apicv-x2apic"/>
<feature policy="require" name="vmx-vpid"/>
<feature policy="require" name="vmx-wbinvd-exit"/>
<feature policy="require" name="vmx-unrestricted-guest"/>
<feature policy="require" name="vmx-rdrand-exit"/>
<feature policy="require" name="vmx-invpcid-exit"/>
<feature policy="require" name="vmx-vmfunc"/>
<feature policy="require" name="vmx-shadow-vmcs"/>
<feature policy="require" name="vmx-pml"/>
<feature policy="require" name="vmx-invvpid"/>
<feature policy="require" name="vmx-invvpid-single-addr"/>
<feature policy="require" name="vmx-invvpid-all-context"/>
<feature policy="require" name="vmx-ept-execonly"/>
<feature policy="require" name="vmx-page-walk-4"/>
<feature policy="require" name="vmx-ept-2mb"/>
<feature policy="require" name="vmx-ept-1gb"/>
<feature policy="require" name="vmx-invept"/>
<feature policy="require" name="vmx-eptad"/>
<feature policy="require"
name="vmx-invept-single-context"/>
<feature policy="require" name="vmx-invept-all-context"/>
<feature policy="require" name="vmx-intr-exit"/>
<feature policy="require" name="vmx-nmi-exit"/>
<feature policy="require" name="vmx-vnmi"/>
<feature policy="require" name="vmx-preemption-timer"/>
<feature policy="require" name="vmx-vintr-pending"/>
<feature policy="require" name="vmx-tsc-offset"/>
<feature policy="require" name="vmx-hlt-exit"/>
<feature policy="require" name="vmx-invlpg-exit"/>
<feature policy="require" name="vmx-mwait-exit"/>
<feature policy="require" name="vmx-rdpmc-exit"/>
<feature policy="require" name="vmx-rdtsc-exit"/>
<feature policy="require" name="vmx-cr3-load-noexit"/>
<feature policy="require" name="vmx-cr3-store-noexit"/>
<feature policy="require" name="vmx-cr8-load-exit"/>
<feature policy="require" name="vmx-cr8-store-exit"/>
<feature policy="require" name="vmx-flexpriority"/>
<feature policy="require" name="vmx-vnmi-pending"/>
<feature policy="require" name="vmx-movdr-exit"/>
<feature policy="require" name="vmx-io-exit"/>
<feature policy="require" name="vmx-io-bitmap"/>
<feature policy="require" name="vmx-mtf"/>
<feature policy="require" name="vmx-msr-bitmap"/>
<feature policy="require" name="vmx-monitor-exit"/>
<feature policy="require" name="vmx-pause-exit"/>
<feature policy="require" name="vmx-secondary-ctls"/>
<feature policy="require" name="vmx-exit-nosave-debugctl"/>
<feature policy="require"
name="vmx-exit-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-exit-ack-intr"/>
<feature policy="require" name="vmx-exit-save-pat"/>
<feature policy="require" name="vmx-exit-load-pat"/>
<feature policy="require" name="vmx-exit-save-efer"/>
<feature policy="require" name="vmx-exit-load-efer"/>
<feature policy="require"
name="vmx-exit-save-preemption-timer"/>
<feature policy="require"
name="vmx-entry-noload-debugctl"/>
<feature policy="require" name="vmx-entry-ia32e-mode"/>
<feature policy="require"
name="vmx-entry-load-perf-global-ctrl"/>
<feature policy="require" name="vmx-entry-load-pat"/>
<feature policy="require" name="vmx-entry-load-efer"/>
<feature policy="require" name="vmx-eptp-switching"/>
<feature policy="disable" name="vmx-apicv-register"/>
<feature policy="disable" name="vmx-apicv-vid"/>
<feature policy="disable" name="vmx-posted-intr"/>
</cpu>
8:37 p.m.
Hi,
So it seems we have a bug regarding the new vmx features when computing
baseline CPU model. But it seems it's not the only thing...
On Fri, May 24, 2024 at 01:20:32 +0800, d tbsky wrote:
Next test I add "vmx" feature to vm config(so it will look
lik RHEL
9.3 config). the xml are the same at both cpu. live migration failed
at both direction and error message are the same: "guest CPU doesn't
match specification: missing features:
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr". we can see these
features were disabled at running xml.
running xml below:
<cpu mode="custom" match="exact" check="full">
<model fallback="forbid">Haswell-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" dies="1" clusters="1"
cores="1" threads="1"/>
<maxphysaddr mode="passthrough" limit="39"/>
...
<feature policy="disable"
name="vmx-apicv-register"/>
<feature policy="disable" name="vmx-apicv-vid"/>
...
<feature policy="disable"
name="vmx-posted-intr"/>
...
</cpu>
Oh, are you saying that a domain with such active XML cannot be migrated
and you see these three features reported as missing? Could you please
provide debug logs from virtqemud (or libvirtd if you're running
monolithic daemon) from both sides of migration covering the failed
attempt? See https://libvirt.org/kbase/debuglogs.html for steps required
to enable debug logs.
Thanks,
Jirka
11:55 p.m.
Hi:
Jiri Denemark <jdenemar(a)redhat.com>
Oh, are you saying that a domain with such active XML cannot be
migrated
and you see these three features reported as missing? Could you please
provide debug logs from virtqemud (or libvirtd if you're running
monolithic daemon) from both sides of migration covering the failed
attempt? See https://libvirt.org/kbase/debuglogs.html for steps required
to enable debug logs.
The logs are a little big. so I put them here
https://drive.google.com/drive/folders/12yZDhP0rKi-47fqljNbtTE7AuCtjPIiY
I tried to migrate from i3 to xeon, then xeon to i3. so I have 4 log
files to record the migration.
There were some other running vms. so please check the vm "kvm-6"
which is the testing vm.
7:20 p.m.
Hi.
On Tue, May 28, 2024 at 23:55:16 +0800, d tbsky wrote:
Jiri Denemark <jdenemar(a)redhat.com>
> Oh, are you saying that a domain with such active XML cannot be migrated
> and you see these three features reported as missing? Could you please
> provide debug logs from virtqemud (or libvirtd if you're running
> monolithic daemon) from both sides of migration covering the failed
> attempt? See https://libvirt.org/kbase/debuglogs.html for steps required
> to enable debug logs.
The logs are a little big. so I put them here
https://drive.google.com/drive/folders/12yZDhP0rKi-47fqljNbtTE7AuCtjPIiY
I tried to migrate from i3 to xeon, then xeon to i3. so I have 4 log
files to record the migration.
There were some other running vms. so please check the vm "kvm-6"
which is the testing vm.
Thanks a lot for the logs. I think I know what the problem is and I'm
trying to reproduce locally and I hope to have a patch ready soon.
Jirka
4:30 p.m.
On Fri, May 31, 2024 at 13:20:49 +0200, Jiri Denemark wrote:
Hi.
On Tue, May 28, 2024 at 23:55:16 +0800, d tbsky wrote:
> Jiri Denemark <jdenemar(a)redhat.com>
> > Oh, are you saying that a domain with such active XML cannot be migrated
> > and you see these three features reported as missing? Could you please
> > provide debug logs from virtqemud (or libvirtd if you're running
> > monolithic daemon) from both sides of migration covering the failed
> > attempt? See https://libvirt.org/kbase/debuglogs.html for steps required
> > to enable debug logs.
>
> The logs are a little big. so I put them here
> https://drive.google.com/drive/folders/12yZDhP0rKi-47fqljNbtTE7AuCtjPIiY
> I tried to migrate from i3 to xeon, then xeon to i3. so I have 4 log
> files to record the migration.
> There were some other running vms. so please check the vm "kvm-6"
> which is the testing vm.
Thanks a lot for the logs. I think I know what the problem is and I'm
trying to reproduce locally and I hope to have a patch ready soon.
This should be fixed by
https://gitlab.com/libvirt/libvirt/-/commit/e622970c8785ec1f7e142d72f792d...
and it will be included in the 10.5.0 release next week.
Jirka
5:55 p.m.
Jiri Denemark <jdenemar(a)redhat.com>
> Thanks a lot for the logs. I think I know what the problem is
and I'm
> trying to reproduce locally and I hope to have a patch ready soon.
This should be fixed by
https://gitlab.com/libvirt/libvirt/-/commit/e622970c8785ec1f7e142d72f792d...
and it will be included in the 10.5.0 release next week.
Jirka
Thanks a lot for your kind help!!
74
days inactive
110
days old
9 comments
2 participants
participants (2)
-
d tbsky -
Jiri Denemark