cpu vmx migration issue
Hi: I update our RHEL9 system to RHEL 9.4, which brings libvirt 10.0. I try to calculate the cpu baseline for our two-node cluster with command "virsh domcapabilities" then "virsh hypervisor-cpu-baseline --migratable". the result has many cpu features begin with "vmx". the test cluster has cpu "intel E3-1280 V3" and "intel I3-9100F". when I try live migrate vm, it failed and told me "guest CPU doesn't match specification: missing features: vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr". at another cluster with cpu " Intel i5-2520M" and "Intel i7-9750H" the migration works fine for the calculated cpu result. although there are still many "vmx" cpu features in the result. if I delete all these vmx features, then migration works fine for both cluster, like old days. I wonder what's the benefit to expose these vmx features to guest if I don't do any nested virtualization. is it ok the drop all these vmx cpu features for a guest? thanks a lot for help!
Hi. On Mon, May 20, 2024 at 21:04:59 +0800, d tbsky wrote:
Hi: I update our RHEL9 system to RHEL 9.4, which brings libvirt 10.0. I try to calculate the cpu baseline for our two-node cluster with command "virsh domcapabilities" then "virsh hypervisor-cpu-baseline --migratable". the result has many cpu features begin with "vmx".
the test cluster has cpu "intel E3-1280 V3" and "intel I3-9100F". when I try live migrate vm, it failed and told me "guest CPU doesn't match specification: missing features: vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr".
Interesting, hypervisor-cpu-baseline is supposed to provide an intersection of both CPU models. I wonder whether we have a bug somewhere, could you share the host-model definition from virsh domcapabilities from both hosts? The easiest way is running $ virsh domcapabilities | xmllint --xpath '//cpu/mode[@name="host-model"]' -
at another cluster with cpu " Intel i5-2520M" and "Intel i7-9750H" the migration works fine for the calculated cpu result. although there are still many "vmx" cpu features in the result.
if I delete all these vmx features, then migration works fine for both cluster, like old days.
I wonder what's the benefit to expose these vmx features to guest if I don't do any nested virtualization.
Well, they were always exposed by QEMU, libvirt just added support for them and thus they are visible in the XMLs as well.
is it ok the drop all these vmx cpu features for a guest? thanks a lot for help!
Sure, unless you specifically need some of them, you can drop them. They will be added to the XML anyway during VM startup. Jirka
Hi: Jiri Denemark <jdenemar@redhat.com>
Interesting, hypervisor-cpu-baseline is supposed to provide an intersection of both CPU models. I wonder whether we have a bug somewhere, could you share the host-model definition from virsh domcapabilities from both hosts? The easiest way is running
$ virsh domcapabilities | xmllint --xpath '//cpu/mode[@name="host-model"]' -
the result are below: for cpu i3-9100F: <mode name="host-model" supported="yes"> <model fallback="forbid">Skylake-Client-IBRS</model> <vendor>Intel</vendor> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="ss"/> <feature policy="require" name="vmx"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="clflushopt"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaves"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="invtsc"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="rsba"/> <feature policy="require" name="pschange-mc-no"/> <feature policy="require" name="vmx-ins-outs"/> <feature policy="require" name="vmx-true-ctls"/> <feature policy="require" name="vmx-store-lma"/> <feature policy="require" name="vmx-activity-hlt"/> <feature policy="require" name="vmx-activity-wait-sipi"/> <feature policy="require" name="vmx-vmwrite-vmexit-fields"/> <feature policy="require" name="vmx-apicv-xapic"/> <feature policy="require" name="vmx-ept"/> <feature policy="require" name="vmx-desc-exit"/> <feature policy="require" name="vmx-rdtscp-exit"/> <feature policy="require" name="vmx-apicv-x2apic"/> <feature policy="require" name="vmx-vpid"/> <feature policy="require" name="vmx-wbinvd-exit"/> <feature policy="require" name="vmx-unrestricted-guest"/> <feature policy="require" name="vmx-rdrand-exit"/> <feature policy="require" name="vmx-invpcid-exit"/> <feature policy="require" name="vmx-vmfunc"/> <feature policy="require" name="vmx-shadow-vmcs"/> <feature policy="require" name="vmx-rdseed-exit"/> <feature policy="require" name="vmx-pml"/> <feature policy="require" name="vmx-xsaves"/> <feature policy="require" name="vmx-invvpid"/> <feature policy="require" name="vmx-invvpid-single-addr"/> <feature policy="require" name="vmx-invvpid-all-context"/> <feature policy="require" name="vmx-ept-execonly"/> <feature policy="require" name="vmx-page-walk-4"/> <feature policy="require" name="vmx-ept-2mb"/> <feature policy="require" name="vmx-ept-1gb"/> <feature policy="require" name="vmx-invept"/> <feature policy="require" name="vmx-eptad"/> <feature policy="require" name="vmx-invept-single-context"/> <feature policy="require" name="vmx-invept-all-context"/> <feature policy="require" name="vmx-intr-exit"/> <feature policy="require" name="vmx-nmi-exit"/> <feature policy="require" name="vmx-vnmi"/> <feature policy="require" name="vmx-preemption-timer"/> <feature policy="require" name="vmx-vintr-pending"/> <feature policy="require" name="vmx-tsc-offset"/> <feature policy="require" name="vmx-hlt-exit"/> <feature policy="require" name="vmx-invlpg-exit"/> <feature policy="require" name="vmx-mwait-exit"/> <feature policy="require" name="vmx-rdpmc-exit"/> <feature policy="require" name="vmx-rdtsc-exit"/> <feature policy="require" name="vmx-cr3-load-noexit"/> <feature policy="require" name="vmx-cr3-store-noexit"/> <feature policy="require" name="vmx-cr8-load-exit"/> <feature policy="require" name="vmx-cr8-store-exit"/> <feature policy="require" name="vmx-flexpriority"/> <feature policy="require" name="vmx-vnmi-pending"/> <feature policy="require" name="vmx-movdr-exit"/> <feature policy="require" name="vmx-io-exit"/> <feature policy="require" name="vmx-io-bitmap"/> <feature policy="require" name="vmx-mtf"/> <feature policy="require" name="vmx-msr-bitmap"/> <feature policy="require" name="vmx-monitor-exit"/> <feature policy="require" name="vmx-pause-exit"/> <feature policy="require" name="vmx-secondary-ctls"/> <feature policy="require" name="vmx-exit-nosave-debugctl"/> <feature policy="require" name="vmx-exit-load-perf-global-ctrl"/> <feature policy="require" name="vmx-exit-ack-intr"/> <feature policy="require" name="vmx-exit-save-pat"/> <feature policy="require" name="vmx-exit-load-pat"/> <feature policy="require" name="vmx-exit-save-efer"/> <feature policy="require" name="vmx-exit-load-efer"/> <feature policy="require" name="vmx-exit-save-preemption-timer"/> <feature policy="require" name="vmx-exit-clear-bndcfgs"/> <feature policy="require" name="vmx-entry-noload-debugctl"/> <feature policy="require" name="vmx-entry-ia32e-mode"/> <feature policy="require" name="vmx-entry-load-perf-global-ctrl"/> <feature policy="require" name="vmx-entry-load-pat"/> <feature policy="require" name="vmx-entry-load-efer"/> <feature policy="require" name="vmx-entry-load-bndcfgs"/> <feature policy="require" name="vmx-eptp-switching"/> <feature policy="disable" name="hle"/> <feature policy="disable" name="rtm"/> </mode> for cpu E3-1280 V3: <mode name="host-model" supported="yes"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="vme"/> <feature policy="require" name="ss"/> <feature policy="require" name="vmx"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="f16c"/> <feature policy="require" name="rdrand"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="arat"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaveopt"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="abm"/> <feature policy="require" name="invtsc"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="pschange-mc-no"/> <feature policy="require" name="gds-no"/> <feature policy="require" name="vmx-ins-outs"/> <feature policy="require" name="vmx-true-ctls"/> <feature policy="require" name="vmx-store-lma"/> <feature policy="require" name="vmx-activity-hlt"/> <feature policy="require" name="vmx-activity-wait-sipi"/> <feature policy="require" name="vmx-vmwrite-vmexit-fields"/> <feature policy="require" name="vmx-apicv-xapic"/> <feature policy="require" name="vmx-ept"/> <feature policy="require" name="vmx-desc-exit"/> <feature policy="require" name="vmx-rdtscp-exit"/> <feature policy="require" name="vmx-apicv-x2apic"/> <feature policy="require" name="vmx-vpid"/> <feature policy="require" name="vmx-wbinvd-exit"/> <feature policy="require" name="vmx-unrestricted-guest"/> <feature policy="require" name="vmx-rdrand-exit"/> <feature policy="require" name="vmx-invpcid-exit"/> <feature policy="require" name="vmx-vmfunc"/> <feature policy="require" name="vmx-shadow-vmcs"/> <feature policy="require" name="vmx-pml"/> <feature policy="require" name="vmx-invvpid"/> <feature policy="require" name="vmx-invvpid-single-addr"/> <feature policy="require" name="vmx-invvpid-all-context"/> <feature policy="require" name="vmx-ept-execonly"/> <feature policy="require" name="vmx-page-walk-4"/> <feature policy="require" name="vmx-ept-2mb"/> <feature policy="require" name="vmx-ept-1gb"/> <feature policy="require" name="vmx-invept"/> <feature policy="require" name="vmx-eptad"/> <feature policy="require" name="vmx-invept-single-context"/> <feature policy="require" name="vmx-invept-all-context"/> <feature policy="require" name="vmx-intr-exit"/> <feature policy="require" name="vmx-nmi-exit"/> <feature policy="require" name="vmx-vnmi"/> <feature policy="require" name="vmx-preemption-timer"/> <feature policy="require" name="vmx-vintr-pending"/> <feature policy="require" name="vmx-tsc-offset"/> <feature policy="require" name="vmx-hlt-exit"/> <feature policy="require" name="vmx-invlpg-exit"/> <feature policy="require" name="vmx-mwait-exit"/> <feature policy="require" name="vmx-rdpmc-exit"/> <feature policy="require" name="vmx-rdtsc-exit"/> <feature policy="require" name="vmx-cr3-load-noexit"/> <feature policy="require" name="vmx-cr3-store-noexit"/> <feature policy="require" name="vmx-cr8-load-exit"/> <feature policy="require" name="vmx-cr8-store-exit"/> <feature policy="require" name="vmx-flexpriority"/> <feature policy="require" name="vmx-vnmi-pending"/> <feature policy="require" name="vmx-movdr-exit"/> <feature policy="require" name="vmx-io-exit"/> <feature policy="require" name="vmx-io-bitmap"/> <feature policy="require" name="vmx-mtf"/> <feature policy="require" name="vmx-msr-bitmap"/> <feature policy="require" name="vmx-monitor-exit"/> <feature policy="require" name="vmx-pause-exit"/> <feature policy="require" name="vmx-secondary-ctls"/> <feature policy="require" name="vmx-exit-nosave-debugctl"/> <feature policy="require" name="vmx-exit-load-perf-global-ctrl"/> <feature policy="require" name="vmx-exit-ack-intr"/> <feature policy="require" name="vmx-exit-save-pat"/> <feature policy="require" name="vmx-exit-load-pat"/> <feature policy="require" name="vmx-exit-save-efer"/> <feature policy="require" name="vmx-exit-load-efer"/> <feature policy="require" name="vmx-exit-save-preemption-timer"/> <feature policy="require" name="vmx-entry-noload-debugctl"/> <feature policy="require" name="vmx-entry-ia32e-mode"/> <feature policy="require" name="vmx-entry-load-perf-global-ctrl"/> <feature policy="require" name="vmx-entry-load-pat"/> <feature policy="require" name="vmx-entry-load-efer"/> <feature policy="require" name="vmx-eptp-switching"/> </mode>
Well, they were always exposed by QEMU, libvirt just added support for them and thus they are visible in the XMLs as well.
indeed. I check the guest machine via "cat /proc/cpuinfo" and vmx is still there even though I deleted these vmx lines.
is it ok the drop all these vmx cpu features for a guest? thanks a lot for help!
Sure, unless you specifically need some of them, you can drop them. They will be added to the XML anyway during VM startup.
indeed. i try to run "virsh dumpxml" for a running vm and I saw many vmx lines. so I got confused. is "virsh dumpxml" supposed to be the same as "hypervisor-cpu-baseline" result? PS: according to redhat document(https://access.redhat.com/documentation/zh-tw/red_hat_enterprise_linux/9/htm...), it use "hypervisor-cpu-baseline" without "--migratable". it will bring in "invtsc" cpu feature. I can never live migrate with the feature.
On Wed, May 22, 2024 at 15:59:36 +0800, d tbsky wrote:
Hi:
Jiri Denemark <jdenemar@redhat.com>
Interesting, hypervisor-cpu-baseline is supposed to provide an intersection of both CPU models. I wonder whether we have a bug somewhere, could you share the host-model definition from virsh domcapabilities from both hosts? The easiest way is running
$ virsh domcapabilities | xmllint --xpath '//cpu/mode[@name="host-model"]' -
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr
the result are below: for cpu i3-9100F:
<mode name="host-model" supported="yes"> <model fallback="forbid">Skylake-Client-IBRS</model> <vendor>Intel</vendor> ... </mode>
for cpu E3-1280 V3:
<mode name="host-model" supported="yes"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> ... </mode>
Thanks, neither CPU model contains any of the features migration is later complaining about (vmx-apicv-register, vmx-apicv-vid, vmx-posted-intr). But they are all included in the Haswell-noTSX-IBRS model while missing in Skylake-Client-IBRS. So I guess if you use the baseline CPU model and start a domain using it on E3-1280 V3 the features might be enabled by QEMU (because the baseline would not explicitly disable them) and thus migrating to i3-9100F would result in such error. Technically the error is correct as the guest could expect the features to be present and try to use them after migration, which would likely cause the guest to crash. On the other hand the computed baseline CPU model would be wrong if my theory is correct. So could you also provide (1) the output of "virsh hypervisor-cpu-baseline" and (2) the <cpu> element from the domain XML before it is started (3) and once it is running on the host from which migration fails? An easy way to extract the <cpu> element from domain XML is: virsh dumpxml nest | xmllint --xpath '/domain/cpu' -
is it ok the drop all these vmx cpu features for a guest? thanks a lot for help!
Sure, unless you specifically need some of them, you can drop them. They will be added to the XML anyway during VM startup.
That said, it may not be safe because of the reason I explained above, but as long as it works, it can be used as a workaround.
indeed. i try to run "virsh dumpxml" for a running vm and I saw many vmx lines. so I got confused. is "virsh dumpxml" supposed to be the same as "hypervisor-cpu-baseline" result?
Due to compatibility reasons libvirt's definition of specific CPU models may differ from the definition QEMU uses so once a domain gets started we query QEMU for the features that were actually enabled (as the set may differ from what we asked for) and updates the domain XML accordingly. Ideally when using a CPU definition computed by hypervisor-cpu-baseline there should be no features added in domain XML (virsh dumpxml) once the domain starts. In you case (as long as my theory is correct) features that we have in our definition of Haswell-noTSX-IBRS are added once domain starts and those features are not available on the second host, which should never happen as such features should have been explicitly disabled in the baseline CPU.
PS: according to redhat document(https://access.redhat.com/documentation/zh-tw/red_hat_enterprise_linux/9/htm...), it use "hypervisor-cpu-baseline" without "--migratable". it will bring in "invtsc" cpu feature. I can never live migrate with the feature.
Yes, without --migratable some features may be included in the result that would block migration. They don't have to though, it depends the common set of features among the hosts for which baseline is computed. And invtsc is not the only non-migratable feature. In other words, using --migratable for hypervisor-cpu-baseline is usually a good idea. Jirka
Hi:
Jiri Denemark <jdenemar@redhat.com> So could you also provide (1) the output of "virsh hypervisor-cpu-baseline" and (2) the <cpu> element from the domain XML before it is started (3) and once it is running on the host from which migration fails?
sorry when I checked the vm with your theory(qemu will enable vmx automatically) in my last post, I wrongly checked a vm which is still using RHEL 9.3 config. RHEL 9.3 config has "vmx" feature but not other "vmx-blabla." sub features. When the vm config has "vmx" feature, "virsh dumpxml" a running vm will bring all these "vmx-blabla" sub feature lines at RHEL 9.4. but if I delete the "vmx" feature from the vm config, then qemu won't enable any "vmx" related feature. "vmx" doesn't exist in guest vm or "virsh dumpxml". and migration works as expected. xml are the same when running at E3-1280-V3 and I3-9100F. the only difference between shutdown/running xml is 'check="partial' and 'check="full"'.
virsh dumpxml my-vm | xmllint --xpath '/domain/cpu' - <cpu mode="custom" match="exact" check="full"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" dies="1" clusters="1" cores="2" threads="1"/> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="vme"/> <feature policy="require" name="ss"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="f16c"/> <feature policy="require" name="rdrand"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="arat"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaveopt"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="abm"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="pschange-mc-no"/> </cpu>
Next test I add "vmx" feature to vm config(so it will look lik RHEL 9.3 config). the xml are the same at both cpu. live migration failed at both direction and error message are the same: "guest CPU doesn't match specification: missing features: vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr". we can see these features were disabled at running xml. shutdown xml below: <cpu mode="custom" match="exact" check="partial"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" dies="1" clusters="1" cores="1" threads="1"/> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="vmx"/> <feature policy="require" name="vme"/> <feature policy="require" name="ss"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="f16c"/> <feature policy="require" name="rdrand"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="arat"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaveopt"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="abm"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="pschange-mc-no"/> </cpu> running xml below: <cpu mode="custom" match="exact" check="full"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" dies="1" clusters="1" cores="1" threads="1"/> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="vmx"/> <feature policy="require" name="vme"/> <feature policy="require" name="ss"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="f16c"/> <feature policy="require" name="rdrand"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="arat"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaveopt"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="abm"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="pschange-mc-no"/> <feature policy="require" name="vmx-ins-outs"/> <feature policy="require" name="vmx-true-ctls"/> <feature policy="require" name="vmx-store-lma"/> <feature policy="require" name="vmx-activity-hlt"/> <feature policy="require" name="vmx-vmwrite-vmexit-fields"/> <feature policy="require" name="vmx-apicv-xapic"/> <feature policy="require" name="vmx-ept"/> <feature policy="require" name="vmx-desc-exit"/> <feature policy="require" name="vmx-rdtscp-exit"/> <feature policy="require" name="vmx-apicv-x2apic"/> <feature policy="require" name="vmx-vpid"/> <feature policy="require" name="vmx-wbinvd-exit"/> <feature policy="require" name="vmx-unrestricted-guest"/> <feature policy="disable" name="vmx-apicv-register"/> <feature policy="disable" name="vmx-apicv-vid"/> <feature policy="require" name="vmx-rdrand-exit"/> <feature policy="require" name="vmx-invpcid-exit"/> <feature policy="require" name="vmx-vmfunc"/> <feature policy="require" name="vmx-shadow-vmcs"/> <feature policy="require" name="vmx-invvpid"/> <feature policy="require" name="vmx-invvpid-single-addr"/> <feature policy="require" name="vmx-invvpid-all-context"/> <feature policy="require" name="vmx-ept-execonly"/> <feature policy="require" name="vmx-page-walk-4"/> <feature policy="require" name="vmx-ept-2mb"/> <feature policy="require" name="vmx-ept-1gb"/> <feature policy="require" name="vmx-invept"/> <feature policy="require" name="vmx-eptad"/> <feature policy="require" name="vmx-invept-single-context"/> <feature policy="require" name="vmx-invept-all-context"/> <feature policy="require" name="vmx-intr-exit"/> <feature policy="require" name="vmx-nmi-exit"/> <feature policy="require" name="vmx-vnmi"/> <feature policy="require" name="vmx-preemption-timer"/> <feature policy="disable" name="vmx-posted-intr"/> <feature policy="require" name="vmx-vintr-pending"/> <feature policy="require" name="vmx-tsc-offset"/> <feature policy="require" name="vmx-hlt-exit"/> <feature policy="require" name="vmx-invlpg-exit"/> <feature policy="require" name="vmx-mwait-exit"/> <feature policy="require" name="vmx-rdpmc-exit"/> <feature policy="require" name="vmx-rdtsc-exit"/> <feature policy="require" name="vmx-cr3-load-noexit"/> <feature policy="require" name="vmx-cr3-store-noexit"/> <feature policy="require" name="vmx-cr8-load-exit"/> <feature policy="require" name="vmx-cr8-store-exit"/> <feature policy="require" name="vmx-flexpriority"/> <feature policy="require" name="vmx-vnmi-pending"/> <feature policy="require" name="vmx-movdr-exit"/> <feature policy="require" name="vmx-io-exit"/> <feature policy="require" name="vmx-io-bitmap"/> <feature policy="require" name="vmx-mtf"/> <feature policy="require" name="vmx-msr-bitmap"/> <feature policy="require" name="vmx-monitor-exit"/> <feature policy="require" name="vmx-pause-exit"/> <feature policy="require" name="vmx-secondary-ctls"/> <feature policy="require" name="vmx-exit-nosave-debugctl"/> <feature policy="require" name="vmx-exit-ack-intr"/> <feature policy="require" name="vmx-exit-save-pat"/> <feature policy="require" name="vmx-exit-load-pat"/> <feature policy="require" name="vmx-exit-save-efer"/> <feature policy="require" name="vmx-exit-load-efer"/> <feature policy="require" name="vmx-exit-save-preemption-timer"/> <feature policy="require" name="vmx-entry-noload-debugctl"/> <feature policy="require" name="vmx-entry-ia32e-mode"/> <feature policy="require" name="vmx-entry-load-pat"/> <feature policy="require" name="vmx-entry-load-efer"/> <feature policy="require" name="vmx-eptp-switching"/> </cpu> the final test I try to use "hypervisor-cpu-baseline --migratable " result. again the xml are the same at both cpu. the difference between shutdown xml/running xml is the last three lines. failed cpu features were disabled at running xml automatically . shutdown xml below: <cpu mode="custom" match="exact" check="partial"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" dies="1" clusters="1" cores="1" threads="1"/> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="vme"/> <feature policy="require" name="ss"/> <feature policy="require" name="vmx"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="f16c"/> <feature policy="require" name="rdrand"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="arat"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaveopt"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="abm"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="pschange-mc-no"/> <feature policy="require" name="vmx-ins-outs"/> <feature policy="require" name="vmx-true-ctls"/> <feature policy="require" name="vmx-store-lma"/> <feature policy="require" name="vmx-activity-hlt"/> <feature policy="require" name="vmx-activity-wait-sipi"/> <feature policy="require" name="vmx-vmwrite-vmexit-fields"/> <feature policy="require" name="vmx-apicv-xapic"/> <feature policy="require" name="vmx-ept"/> <feature policy="require" name="vmx-desc-exit"/> <feature policy="require" name="vmx-rdtscp-exit"/> <feature policy="require" name="vmx-apicv-x2apic"/> <feature policy="require" name="vmx-vpid"/> <feature policy="require" name="vmx-wbinvd-exit"/> <feature policy="require" name="vmx-unrestricted-guest"/> <feature policy="require" name="vmx-rdrand-exit"/> <feature policy="require" name="vmx-invpcid-exit"/> <feature policy="require" name="vmx-vmfunc"/> <feature policy="require" name="vmx-shadow-vmcs"/> <feature policy="require" name="vmx-pml"/> <feature policy="require" name="vmx-invvpid"/> <feature policy="require" name="vmx-invvpid-single-addr"/> <feature policy="require" name="vmx-invvpid-all-context"/> <feature policy="require" name="vmx-ept-execonly"/> <feature policy="require" name="vmx-page-walk-4"/> <feature policy="require" name="vmx-ept-2mb"/> <feature policy="require" name="vmx-ept-1gb"/> <feature policy="require" name="vmx-invept"/> <feature policy="require" name="vmx-eptad"/> <feature policy="require" name="vmx-invept-single-context"/> <feature policy="require" name="vmx-invept-all-context"/> <feature policy="require" name="vmx-intr-exit"/> <feature policy="require" name="vmx-nmi-exit"/> <feature policy="require" name="vmx-vnmi"/> <feature policy="require" name="vmx-preemption-timer"/> <feature policy="require" name="vmx-vintr-pending"/> <feature policy="require" name="vmx-tsc-offset"/> <feature policy="require" name="vmx-hlt-exit"/> <feature policy="require" name="vmx-invlpg-exit"/> <feature policy="require" name="vmx-mwait-exit"/> <feature policy="require" name="vmx-rdpmc-exit"/> <feature policy="require" name="vmx-rdtsc-exit"/> <feature policy="require" name="vmx-cr3-load-noexit"/> <feature policy="require" name="vmx-cr3-store-noexit"/> <feature policy="require" name="vmx-cr8-load-exit"/> <feature policy="require" name="vmx-cr8-store-exit"/> <feature policy="require" name="vmx-flexpriority"/> <feature policy="require" name="vmx-vnmi-pending"/> <feature policy="require" name="vmx-movdr-exit"/> <feature policy="require" name="vmx-io-exit"/> <feature policy="require" name="vmx-io-bitmap"/> <feature policy="require" name="vmx-mtf"/> <feature policy="require" name="vmx-msr-bitmap"/> <feature policy="require" name="vmx-monitor-exit"/> <feature policy="require" name="vmx-pause-exit"/> <feature policy="require" name="vmx-secondary-ctls"/> <feature policy="require" name="vmx-exit-nosave-debugctl"/> <feature policy="require" name="vmx-exit-load-perf-global-ctrl"/> <feature policy="require" name="vmx-exit-ack-intr"/> <feature policy="require" name="vmx-exit-save-pat"/> <feature policy="require" name="vmx-exit-load-pat"/> <feature policy="require" name="vmx-exit-save-efer"/> <feature policy="require" name="vmx-exit-load-efer"/> <feature policy="require" name="vmx-exit-save-preemption-timer"/> <feature policy="require" name="vmx-entry-noload-debugctl"/> <feature policy="require" name="vmx-entry-ia32e-mode"/> <feature policy="require" name="vmx-entry-load-perf-global-ctrl"/> <feature policy="require" name="vmx-entry-load-pat"/> <feature policy="require" name="vmx-entry-load-efer"/> <feature policy="require" name="vmx-eptp-switching"/> </cpu> running xml below: <cpu mode="custom" match="exact" check="full"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" dies="1" clusters="1" cores="1" threads="1"/> <maxphysaddr mode="passthrough" limit="39"/> <feature policy="require" name="vme"/> <feature policy="require" name="ss"/> <feature policy="require" name="vmx"/> <feature policy="require" name="pdcm"/> <feature policy="require" name="f16c"/> <feature policy="require" name="rdrand"/> <feature policy="require" name="hypervisor"/> <feature policy="require" name="arat"/> <feature policy="require" name="tsc_adjust"/> <feature policy="require" name="umip"/> <feature policy="require" name="md-clear"/> <feature policy="require" name="stibp"/> <feature policy="require" name="flush-l1d"/> <feature policy="require" name="arch-capabilities"/> <feature policy="require" name="ssbd"/> <feature policy="require" name="xsaveopt"/> <feature policy="require" name="pdpe1gb"/> <feature policy="require" name="abm"/> <feature policy="require" name="ibpb"/> <feature policy="require" name="ibrs"/> <feature policy="require" name="amd-stibp"/> <feature policy="require" name="amd-ssbd"/> <feature policy="require" name="pschange-mc-no"/> <feature policy="require" name="vmx-ins-outs"/> <feature policy="require" name="vmx-true-ctls"/> <feature policy="require" name="vmx-store-lma"/> <feature policy="require" name="vmx-activity-hlt"/> <feature policy="require" name="vmx-activity-wait-sipi"/> <feature policy="require" name="vmx-vmwrite-vmexit-fields"/> <feature policy="require" name="vmx-apicv-xapic"/> <feature policy="require" name="vmx-ept"/> <feature policy="require" name="vmx-desc-exit"/> <feature policy="require" name="vmx-rdtscp-exit"/> <feature policy="require" name="vmx-apicv-x2apic"/> <feature policy="require" name="vmx-vpid"/> <feature policy="require" name="vmx-wbinvd-exit"/> <feature policy="require" name="vmx-unrestricted-guest"/> <feature policy="require" name="vmx-rdrand-exit"/> <feature policy="require" name="vmx-invpcid-exit"/> <feature policy="require" name="vmx-vmfunc"/> <feature policy="require" name="vmx-shadow-vmcs"/> <feature policy="require" name="vmx-pml"/> <feature policy="require" name="vmx-invvpid"/> <feature policy="require" name="vmx-invvpid-single-addr"/> <feature policy="require" name="vmx-invvpid-all-context"/> <feature policy="require" name="vmx-ept-execonly"/> <feature policy="require" name="vmx-page-walk-4"/> <feature policy="require" name="vmx-ept-2mb"/> <feature policy="require" name="vmx-ept-1gb"/> <feature policy="require" name="vmx-invept"/> <feature policy="require" name="vmx-eptad"/> <feature policy="require" name="vmx-invept-single-context"/> <feature policy="require" name="vmx-invept-all-context"/> <feature policy="require" name="vmx-intr-exit"/> <feature policy="require" name="vmx-nmi-exit"/> <feature policy="require" name="vmx-vnmi"/> <feature policy="require" name="vmx-preemption-timer"/> <feature policy="require" name="vmx-vintr-pending"/> <feature policy="require" name="vmx-tsc-offset"/> <feature policy="require" name="vmx-hlt-exit"/> <feature policy="require" name="vmx-invlpg-exit"/> <feature policy="require" name="vmx-mwait-exit"/> <feature policy="require" name="vmx-rdpmc-exit"/> <feature policy="require" name="vmx-rdtsc-exit"/> <feature policy="require" name="vmx-cr3-load-noexit"/> <feature policy="require" name="vmx-cr3-store-noexit"/> <feature policy="require" name="vmx-cr8-load-exit"/> <feature policy="require" name="vmx-cr8-store-exit"/> <feature policy="require" name="vmx-flexpriority"/> <feature policy="require" name="vmx-vnmi-pending"/> <feature policy="require" name="vmx-movdr-exit"/> <feature policy="require" name="vmx-io-exit"/> <feature policy="require" name="vmx-io-bitmap"/> <feature policy="require" name="vmx-mtf"/> <feature policy="require" name="vmx-msr-bitmap"/> <feature policy="require" name="vmx-monitor-exit"/> <feature policy="require" name="vmx-pause-exit"/> <feature policy="require" name="vmx-secondary-ctls"/> <feature policy="require" name="vmx-exit-nosave-debugctl"/> <feature policy="require" name="vmx-exit-load-perf-global-ctrl"/> <feature policy="require" name="vmx-exit-ack-intr"/> <feature policy="require" name="vmx-exit-save-pat"/> <feature policy="require" name="vmx-exit-load-pat"/> <feature policy="require" name="vmx-exit-save-efer"/> <feature policy="require" name="vmx-exit-load-efer"/> <feature policy="require" name="vmx-exit-save-preemption-timer"/> <feature policy="require" name="vmx-entry-noload-debugctl"/> <feature policy="require" name="vmx-entry-ia32e-mode"/> <feature policy="require" name="vmx-entry-load-perf-global-ctrl"/> <feature policy="require" name="vmx-entry-load-pat"/> <feature policy="require" name="vmx-entry-load-efer"/> <feature policy="require" name="vmx-eptp-switching"/> <feature policy="disable" name="vmx-apicv-register"/> <feature policy="disable" name="vmx-apicv-vid"/> <feature policy="disable" name="vmx-posted-intr"/> </cpu>
Hi, So it seems we have a bug regarding the new vmx features when computing baseline CPU model. But it seems it's not the only thing... On Fri, May 24, 2024 at 01:20:32 +0800, d tbsky wrote:
Next test I add "vmx" feature to vm config(so it will look lik RHEL 9.3 config). the xml are the same at both cpu. live migration failed at both direction and error message are the same: "guest CPU doesn't match specification: missing features: vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr". we can see these features were disabled at running xml.
running xml below: <cpu mode="custom" match="exact" check="full"> <model fallback="forbid">Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" dies="1" clusters="1" cores="1" threads="1"/> <maxphysaddr mode="passthrough" limit="39"/> ... <feature policy="disable" name="vmx-apicv-register"/> <feature policy="disable" name="vmx-apicv-vid"/> ... <feature policy="disable" name="vmx-posted-intr"/> ... </cpu>
Oh, are you saying that a domain with such active XML cannot be migrated and you see these three features reported as missing? Could you please provide debug logs from virtqemud (or libvirtd if you're running monolithic daemon) from both sides of migration covering the failed attempt? See https://libvirt.org/kbase/debuglogs.html for steps required to enable debug logs. Thanks, Jirka
Hi: Jiri Denemark <jdenemar@redhat.com>
Oh, are you saying that a domain with such active XML cannot be migrated and you see these three features reported as missing? Could you please provide debug logs from virtqemud (or libvirtd if you're running monolithic daemon) from both sides of migration covering the failed attempt? See https://libvirt.org/kbase/debuglogs.html for steps required to enable debug logs.
The logs are a little big. so I put them here https://drive.google.com/drive/folders/12yZDhP0rKi-47fqljNbtTE7AuCtjPIiY I tried to migrate from i3 to xeon, then xeon to i3. so I have 4 log files to record the migration. There were some other running vms. so please check the vm "kvm-6" which is the testing vm.
Hi. On Tue, May 28, 2024 at 23:55:16 +0800, d tbsky wrote:
Jiri Denemark <jdenemar@redhat.com>
Oh, are you saying that a domain with such active XML cannot be migrated and you see these three features reported as missing? Could you please provide debug logs from virtqemud (or libvirtd if you're running monolithic daemon) from both sides of migration covering the failed attempt? See https://libvirt.org/kbase/debuglogs.html for steps required to enable debug logs.
The logs are a little big. so I put them here https://drive.google.com/drive/folders/12yZDhP0rKi-47fqljNbtTE7AuCtjPIiY I tried to migrate from i3 to xeon, then xeon to i3. so I have 4 log files to record the migration. There were some other running vms. so please check the vm "kvm-6" which is the testing vm.
Thanks a lot for the logs. I think I know what the problem is and I'm trying to reproduce locally and I hope to have a patch ready soon. Jirka
On Fri, May 31, 2024 at 13:20:49 +0200, Jiri Denemark wrote:
Hi.
On Tue, May 28, 2024 at 23:55:16 +0800, d tbsky wrote:
Jiri Denemark <jdenemar@redhat.com>
Oh, are you saying that a domain with such active XML cannot be migrated and you see these three features reported as missing? Could you please provide debug logs from virtqemud (or libvirtd if you're running monolithic daemon) from both sides of migration covering the failed attempt? See https://libvirt.org/kbase/debuglogs.html for steps required to enable debug logs.
The logs are a little big. so I put them here https://drive.google.com/drive/folders/12yZDhP0rKi-47fqljNbtTE7AuCtjPIiY I tried to migrate from i3 to xeon, then xeon to i3. so I have 4 log files to record the migration. There were some other running vms. so please check the vm "kvm-6" which is the testing vm.
Thanks a lot for the logs. I think I know what the problem is and I'm trying to reproduce locally and I hope to have a patch ready soon.
This should be fixed by https://gitlab.com/libvirt/libvirt/-/commit/e622970c8785ec1f7e142d72f792d89f... and it will be included in the 10.5.0 release next week. Jirka
Jiri Denemark <jdenemar@redhat.com>
Thanks a lot for the logs. I think I know what the problem is and I'm trying to reproduce locally and I hope to have a patch ready soon.
This should be fixed by https://gitlab.com/libvirt/libvirt/-/commit/e622970c8785ec1f7e142d72f792d89f... and it will be included in the 10.5.0 release next week.
Jirka
Thanks a lot for your kind help!!
participants (2)
-
d tbsky -
Jiri Denemark