[libvirt-users] Help With Nested Virtualization
(Copied from my post on the Arch Linux forums: https://bbs.archlinux.org/viewtopic.php?pid=1650650#p1650650) I have a FreeNAS 10 KVM setup via libvirt on my Arch server and I'd like to be able to test out the virtualization features in the nightly FreeNAS 10 builds but the problem is that I can't seem to get VT-x to correctly pass through to the guest. I have followed the [u][url= https://wiki.archlinux.org/index.php/KVM#Nested_virtualization]Nested Virtualization[/url][/u] section of the KVM wiki and I'm sure it's supported and enabled. Here's proof [code] [bran@nas ~]$ sudo systool -m kvm_intel -v | grep nested nested = "Y" [bran@nas ~]$ lscpu|grep Virtualization Virtualization: VT-x[/code] So the host/hardware isn't the problem, I believe the problem lies within libvirt. RedHat says to use [b]copy host CPU configuration[/b] or [b]host-passthrough[/b], with the latter being preferred. If I use the former I get the [b]CMT not supported[/b] error, but if I type in [b]host-passthrough[/b] which according to [u][url= https://bbs.archlinux.org/viewtopic.php?id=214539]this post[/url][/u] should work. When I set [b]host-passthrough[/b] it allows the system to boot up, but when I try to start a guest FreeNAS gives me the error that VT-x instructions aren't available, I have no idea how to check them either since the [b]proc[/b] pseudo-filesystem doesn't exist in BSD. I know for a fact that this isn't a problem with the FreeNAS builds because I've been testing them out for months on my Windows 10 desktop via VMware and nested virtualization works without issue, so it must be an issue with KVM/libvirt. Can someone clue me in on what the issue is? Here's the entire XML config for the FreeNAS VM [code]<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit FreeNAS_10 or other application using the libvirt API. --> <domain type='kvm'> <name>FreeNAS_10</name> <uuid>ea816b85-7685-495a-bc97-28a882f190d7</uuid> <title>FreeNAS v10</title> <description>Nightly Alpha Test Releases</description> <memory unit='KiB'>6340608</memory> <currentMemory unit='KiB'>6340608</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-passthrough'/> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/storage/vm-storage/FreeNAS_Disk1.img'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/storage/vm-storage/FreeNAS_Disk2.img'/> <target dev='vdc' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/storage/vm-storage/FreeNAS_Disk3.img'/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='directsync' io='native'/> <source file='/var/lib/libvirt/images/FreeNAS_10.img'/> <target dev='vde' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:24:5c:08'/> <source bridge='vmbridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='1'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <address type='usb' bus='0' port='2'/> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </memballoon> </devices> </domain>[/code] Thanks, Brandon Golway
On Mon, Aug 29, 2016 at 09:42:17PM -0400, Brandon Golway wrote:
(Copied from my post on the Arch Linux forums: https://bbs.archlinux.org/viewtopic.php?pid=1650650#p1650650)
I have a FreeNAS 10 KVM setup via libvirt on my Arch server and I'd like to be able to test out the virtualization features in the nightly FreeNAS 10 builds but the problem is that I can't seem to get VT-x to correctly pass through to the guest. I have followed the [u][url= https://wiki.archlinux.org/index.php/KVM#Nested_virtualization]Nested Virtualization[/url][/u] section of the KVM wiki and I'm sure it's supported and enabled.
Here's proof [code] [bran@nas ~]$ sudo systool -m kvm_intel -v | grep nested nested = "Y"
[bran@nas ~]$ lscpu|grep Virtualization Virtualization: VT-x[/code]
So the host/hardware isn't the problem, I believe the problem lies within libvirt.
RedHat says to use [b]copy host CPU configuration[/b] or [b]host-passthrough[/b], with the latter being preferred. If I use the former I get the [b]CMT not supported[/b] error, but if I type in [b]host-passthrough[/b] which according to [u][url= https://bbs.archlinux.org/viewtopic.php?id=214539]this post[/url][/u] should work. When I set [b]host-passthrough[/b] it allows the system to boot up, but when I try to start a guest FreeNAS gives me the error that VT-x instructions aren't available, I have no idea how to check them either since the [b]proc[/b] pseudo-filesystem doesn't exist in BSD. I know for a fact that this isn't a problem with the FreeNAS builds because I've been testing them out for months on my Windows 10 desktop via VMware and nested virtualization works without issue, so it must be an issue with KVM/libvirt.
Can someone clue me in on what the issue is?
Here's the entire XML config for the FreeNAS VM
[code]<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit FreeNAS_10 or other application using the libvirt API. -->
Just as I see you grabbed it from the place where it is stored, you should rather instead do 'virsh dumpxml FreeNAS_10', or remove the comment above. Otherwise it steers other users to believing that mangling with internal XML files is safe.
<domain type='kvm'> <name>FreeNAS_10</name> <uuid>ea816b85-7685-495a-bc97-28a882f190d7</uuid> <title>FreeNAS v10</title> <description>Nightly Alpha Test Releases</description> <memory unit='KiB'>6340608</memory> <currentMemory unit='KiB'>6340608</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-passthrough'/>
Although what you did should be enough for the nested virt to work, I have just an idea to try out. I can't try it because I don't have nested virt turned on for the module and don't want to reboot now. But if you have new enough libvirt, you should be able to do: <cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu> Also check that you are not missing the kvm_intel module in the L1 guest kernel and other similar non-nested troubleshooting steps as well. Hope that helps, Martin
On Wed, Aug 31, 2016 at 12:48:33PM +0200, Martin Kletzander wrote:
On Mon, Aug 29, 2016 at 09:42:17PM -0400, Brandon Golway wrote:
(Copied from my post on the Arch Linux forums:
[...]
Can someone clue me in on what the issue is?
Here's the entire XML config for the FreeNAS VM
[code]<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit FreeNAS_10 or other application using the libvirt API. -->
Just as I see you grabbed it from the place where it is stored, you should rather instead do 'virsh dumpxml FreeNAS_10', or remove the comment above. Otherwise it steers other users to believing that mangling with internal XML files is safe.
Yeah, or use the `virt-xml` tool (comes as part of `virt-install` package). To pass-through host CPU to the level-1 guest: $ virt-xml FreeNAS_10 --edit --cpu host-passthrough,clearxml=yes If you issue this while the guest is running, it'll take affect on next reboot.
<domain type='kvm'> <name>FreeNAS_10</name> <uuid>ea816b85-7685-495a-bc97-28a882f190d7</uuid> <title>FreeNAS v10</title> <description>Nightly Alpha Test Releases</description> <memory unit='KiB'>6340608</memory> <currentMemory unit='KiB'>6340608</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-passthrough'/>
Although what you did should be enough for the nested virt to work, I have just an idea to try out. I can't try it because I don't have nested virt turned on for the module and don't want to reboot now.
Assuming you're talking about the physical host, you don't need a reboot of the host :-) You can simply: $ sudo rmmod kvm-intel $ echo "options kvm-intel nested=y" > /etc/modprobe.d/dist.conf $ sudo modprobe kvm-intel
But if you have new enough libvirt, you should be able to do:
<cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu>
Also check that you are not missing the kvm_intel module in the L1 guest kernel and other similar non-nested troubleshooting steps as well.
Yeah. If the L1 guest has the /dev/kvm character device, then he's good: $ file /dev/kvm /dev/kvm: character special (10/232) -- /kashyap
On Thu, Sep 01, 2016 at 12:11:17PM +0200, Kashyap Chamarthy wrote:
On Wed, Aug 31, 2016 at 12:48:33PM +0200, Martin Kletzander wrote:
On Mon, Aug 29, 2016 at 09:42:17PM -0400, Brandon Golway wrote:
(Copied from my post on the Arch Linux forums:
[...]
Can someone clue me in on what the issue is?
Here's the entire XML config for the FreeNAS VM
[code]<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit FreeNAS_10 or other application using the libvirt API. -->
Just as I see you grabbed it from the place where it is stored, you should rather instead do 'virsh dumpxml FreeNAS_10', or remove the comment above. Otherwise it steers other users to believing that mangling with internal XML files is safe.
Yeah, or use the `virt-xml` tool (comes as part of `virt-install` package). To pass-through host CPU to the level-1 guest:
$ virt-xml FreeNAS_10 --edit --cpu host-passthrough,clearxml=yes
If you issue this while the guest is running, it'll take affect on next reboot.
<domain type='kvm'> <name>FreeNAS_10</name> <uuid>ea816b85-7685-495a-bc97-28a882f190d7</uuid> <title>FreeNAS v10</title> <description>Nightly Alpha Test Releases</description> <memory unit='KiB'>6340608</memory> <currentMemory unit='KiB'>6340608</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <vmport state='off'/> </features> <cpu mode='host-passthrough'/>
Although what you did should be enough for the nested virt to work, I have just an idea to try out. I can't try it because I don't have nested virt turned on for the module and don't want to reboot now.
Assuming you're talking about the physical host, you don't need a reboot of the host :-) You can simply:
$ sudo rmmod kvm-intel $ echo "options kvm-intel nested=y" > /etc/modprobe.d/dist.conf $ sudo modprobe kvm-intel
Unless: $ zgrep -i kvm_intel /proc/config.gz CONFIG_KVM_INTEL=y Maybe I could switch to =m when I'm building the next kernel.
But if you have new enough libvirt, you should be able to do:
<cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu>
Also check that you are not missing the kvm_intel module in the L1 guest kernel and other similar non-nested troubleshooting steps as well.
Yeah. If the L1 guest has the /dev/kvm character device, then he's good:
$ file /dev/kvm /dev/kvm: character special (10/232)
-- /kashyap
hi all & @Kayshap does [this handout on snapshots](https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html) by Kashyap Chamarthy (I believe from 2012) still apply to current versions of libvirt-bin & qemu? - ubuntu: 14.04 - qemu: 2.0.0 - libvirt-bin 1.2.2-0ubuntu13.1.17 if not, is there an updated version available somewhere? Right now mainly interested in internal snapshots (which can be taken & removed on a running machine, if I am not mistaken)
On Sat, Sep 03, 2016 at 02:39:11PM +0800, gunnar.wagner@netcologne.de wrote:
hi all & @Kayshap
does [this handout on snapshots](https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html) by Kashyap Chamarthy (I believe from 2012) still apply to current versions of libvirt-bin & qemu?
Yes, as the URL indicates, it's from 2012. And, most of the content there is valid. However, some things have changed, with recent improvements in QEMU and libvirt. E.g. from the "blockcommit" section in the doc, now (from QEMU 2.0) it is possible to merge down the current active layer into base.
- ubuntu: 14.04 - qemu: 2.0.0 - libvirt-bin 1.2.2-0ubuntu13.1.17
if not, is there an updated version available somewhere?
Though I haven't updated that document to reflect current new reality, that details mentioned there are still relevant. However, Eric Blake has done an excellent talk on backing chains, snapshots, and their management at KVM Forum 2015. Here's the talk (slides are in the video description): https://www.youtube.com/watch?v=etIGp12RHRE -- Backing Chain management in QEMU and libvirt by Eric Blake
Right now mainly interested in internal snapshots (which can be taken & removed on a running machine, if I am not mistaken)
Yes, internal snapshots can be performed on a running guest (or offline). Compare and contrast them with external snapshots, to see what works best for your use case. -- /kashyap
participants (4)
-
Brandon Golway -
gunnar.wagner＠netcologne.de -
Kashyap Chamarthy -
Martin Kletzander