[Cc: KVM upstream list.] On Tue, Feb 06, 2018 at 04:11:46PM +0100, Florian Haas wrote:
Hi everyone,
I hope this is the correct list to discuss this issue; please feel free to redirect me otherwise.
I have a nested virtualization setup that looks as follows:
- Host: Ubuntu 16.04, kernel 4.4.0 (an OpenStack Nova compute node) - L0 guest: openSUSE Leap 42.3, kernel 4.4.104-39-default - Nested guest: SLES 12, kernel 3.12.28-4-default
The nested guest is configured with "<type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type>".
This is working just beautifully, except when the L0 guest wakes up from managed save (openstack server resume in OpenStack parlance). Then, in the L0 guest we immediately see this:
[...] # Snip the call trace from Florian. It is here: https://www.redhat.com/archives/libvirt-users/2018-February/msg00014.html
What does fix things, of course, is to switch from the nested guest from KVM to Qemu — but that also makes things significantly slower.
So I'm wondering: is there someone reading this who does run nested KVM and has managed to successfully live-migrate or managed-save? If so, would you be able to share a working host kernel / L0 guest kernel / nested guest kernel combination, or any other hints for tuning the L0 guest to support managed save and live migration?
Following up from our IRC discussion (on #kvm, Freenode). Re-posting my comment here: So I just did a test of 'managedsave' (which is just "save the state of the running VM to a file" in libvirt parlance) of L1, _while_ L2 is running, and I seem to reproduce your case (see the call trace attached). # Ensure L2 (the nested guest) is running on L1. Then, from L0, do # the following: [L0] $ virsh managedsave L1 [L0] $ virsh start L1 --console Result: See the call trace attached to this bug. But L1 goes on to start "fine", and L2 keeps running, too. But things start to seem weird. As in: I try to safely, read-only mount the L2 disk image via libguestfs (by setting export LIBGUESTFS_BACKEND=direct, which uses direct QEMU): `guestfish --ro -a -i ./cirros.qcow2`. It throws the call trace again on the L1 serial console. And the `guestfish` command just sits there forever - L0 (bare metal) Kernel: 4.13.13-300.fc27.x86_64+debug - L1 (guest hypervisor) kernel: 4.11.10-300.fc26.x86_64 - L2 is a CirrOS 3.5 image I can reproduce this at least 3 times, with the above versions. I'm using libvirt 'host-passthrough' for CPU (meaning: '-cpu host' in QEMU parlance) for both L1 and L2. My L0 CPU is: Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz. Thoughts? --- [/me wonders if I'll be asked to reproduce this with newest upstream kernels.] [...] -- /kashyap