On Thu, Feb 08, 2018 at 01:07:33PM +0100, David Hildenbrand wrote: [...]
So to clarify things, could you enumerate the currently known limitations when enabling nesting? I'd be happy to summarize those and add them to the linux-kvm.org FAQ so others are less likely to hit their head on this issue. In particular:
[...] # Snip description of what works in context of migration
- Is https://fedoraproject.org/wiki/How_to_enable_nested_virtualization_in_KVM still accurate in that -cpu host (libvirt "host-passthrough") is the strongly recommended configuration for the L2 guest?
That wiki is a bit outdated. And it is not accurate — if we can just expose the Intel 'vmx' (or AMD 'svm') CPU feature flag to the L2 guest, that should be sufficient. No need for a full passthrough. That above document should definitely be modified to add more verbiage comparing 'host-passthrough' vs. 'host-model' vs. custom CPU.
- If so, are there any recommendations for how to configure the L1 guest with regard to CPU model?
You have to indicate the VMX feature to your L1 ("nested hypervisor"), that is usually automatically done by using the "host-passthrough" or "host-model" value. If you're using a custom CPU model, you have to enable it explicitly.
- Is live migration with nested guests _always_ expected to break on all architectures, and if not, which are safe?
x86 VMX: running nested guests works, migrating nested hypervisors does not work
x86 SVM: running nested guests works, migrating nested hypervisor does not work (somebody correct me if I'm wrong)
s390x: running nested guests works, migrating nested hypervisors works
power: running nested guests works only via KVM-PR ("trap and emulate"). migrating nested hypervisors therefore works. But we are not using hardware virtualization for L1->L2. (my latest status)
arm: running nested guests is in the works (my latest status), migration is therefore also not possible.
That's a great summary.
- Idem, for savevm/loadvm?
savevm/loadvm is not expected to work correctly on an L1 if it is running L2 guests. It should work on L2 however.
Yes, that works as intended.
- With regard to the problem that Kashyap and I (and Dennis, the kernel.org bugzilla reporter) are describing, is this expected to work any better on AMD CPUs? (All reports are on Intel)
No, remeber that they are also still missing migration support of the nested SVM state.
Right. I partly mixed up migration of L1-running-L2 (which doesn't fly for reasons David already explained) vs. migrating L2 (which works).
- Do you expect nested virtualization functionality to be adversely affected by KPTI and/or other Meltdown/Spectre mitigation patches?
Not an expert on this. I think it should be affected in a similar way as ordinary guests :)
Kashyap, can you think of any other limitations that would benefit from improved documentation?
We should certainly document what I have summaries here properly at a central palce!
Yeah, agreed. Also, when documentation in context of nested, it'd be useful to explicitly spell out what works or doesn't work at each level — e.g. L2 can be migrated to a destination L1 just fine; mirating an L1-running-L2 to a destination L0 will be in dodgy waters for reasons X, etc. [...] -- /kashyap