[libvirt] [PATCH v1 0/3] libxl: nestedhvm support
From: Wim ten Have <wim.ten.have@oracle.com> This patch enhances host-passthrough capability to advertise the required vendor CPU virtualization feature which will be used to enable 'nestedhvm' in the libxl driver. For Intel virtualization (VT-x) .. <cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu> For AMD virtualization (AMD-V) .. <cpu mode='host-passthrough'> <feature policy='require' name='svm'/> </cpu> Joao Martins (2): libxl: set nestedhvm when vmx|svm is required xenconfig: add conversion of domxml nestedhvm Wim ten Have (1): xlconfigtest: add test and data for 'nestedhvm' support src/libxl/libxl_conf.c | 9 ++++ src/xenconfig/xen_xl.c | 41 +++++++++++++++++ tests/xlconfigdata/test-fullvirt-nestedhvm.cfg | 26 +++++++++++ tests/xlconfigdata/test-fullvirt-nestedhvm.xml | 61 ++++++++++++++++++++++++++ tests/xlconfigtest.c | 1 + 5 files changed, 138 insertions(+) create mode 100644 tests/xlconfigdata/test-fullvirt-nestedhvm.cfg create mode 100644 tests/xlconfigdata/test-fullvirt-nestedhvm.xml -- 2.9.3
From: Joao Martins <joao.m.martins@oracle.com> nestedhvm is the option on Xen 4.4+ which enables nested virtualization on Xen. We set this field when one of the following features is included in a host-passthrough CPU element: <feature policy='require' name='vmx'/> or: <feature policy='require' name='svm'/> Signed-off-by: Joao Martins <joao.m.martins@oracle.com> Signed-off-by: Wim ten Have <wim.ten.have@oracle.com> --- src/libxl/libxl_conf.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/libxl/libxl_conf.c b/src/libxl/libxl_conf.c index f5b788b..8c4c511 100644 --- a/src/libxl/libxl_conf.c +++ b/src/libxl/libxl_conf.c @@ -374,6 +374,15 @@ libxlMakeDomBuildInfo(virDomainDefPtr def, def->features[VIR_DOMAIN_FEATURE_ACPI] == VIR_TRISTATE_SWITCH_ON); + if (def->cpu && def->cpu->nfeatures) { + for (i = 0; i < def->cpu->nfeatures; i++) { + if (def->cpu->features[i].policy == VIR_CPU_FEATURE_REQUIRE && + (STREQ(def->cpu->features[i].name, "vmx") || + STREQ(def->cpu->features[i].name, "svm"))) + libxl_defbool_set(&b_info->u.hvm.nested_hvm, true); + } + } + if (def->nsounds > 0) { /* * Use first sound device. man xl.cfg(5) describes soundhw as -- 2.9.3
From: Joao Martins <joao.m.martins@oracle.com> In other words, on configurations containing (within its CPU element) one of the following for a host-passthrough mode: <feature policy='require' name='vmx'/> <feature policy='require' name='svm'/> It will then generate (or parse) for nestedhvm=1 in/from xl format. Signed-off-by: Joao Martins <joao.m.martins@oracle.com> Signed-off-by: Wim ten Have <wim.ten.have@oracle.com> --- src/xenconfig/xen_xl.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/src/xenconfig/xen_xl.c b/src/xenconfig/xen_xl.c index 74f68b3..738dcd0 100644 --- a/src/xenconfig/xen_xl.c +++ b/src/xenconfig/xen_xl.c @@ -106,6 +106,7 @@ xenParseXLOS(virConfPtr conf, virDomainDefPtr def, virCapsPtr caps) if (def->os.type == VIR_DOMAIN_OSTYPE_HVM) { const char *bios; const char *boot; + int val; if (xenConfigGetString(conf, "bios", &bios, NULL) < 0) return -1; @@ -164,6 +165,35 @@ xenParseXLOS(virConfPtr conf, virDomainDefPtr def, virCapsPtr caps) } def->os.nBootDevs++; } + + if (xenConfigGetBool(conf, "nestedhvm", &val, 0) < 0) { + return -1; + } else if (val) { + virCPUDefPtr cpu = NULL; + + if (VIR_ALLOC(cpu) < 0) + return -1; + + if (VIR_ALLOC_N(cpu->features, 1) < 0) + goto cleanup; + + cpu->features[0].policy = VIR_CPU_FEATURE_REQUIRE; + if (VIR_STRDUP(cpu->features[0].name, "vmx") < 0) + goto cleanup; + + cpu->nfeatures = cpu->nfeatures_max = 1; + cpu->mode = VIR_CPU_MODE_HOST_PASSTHROUGH; + cpu->type = VIR_CPU_TYPE_GUEST; + def->cpu = cpu; + cpu = NULL; + + cleanup: + if (cpu) { + VIR_FREE(cpu->features); + VIR_FREE(cpu); + return -1; + } + } } else { if (xenConfigCopyStringOpt(conf, "bootloader", &def->os.bootloader) < 0) return -1; @@ -897,6 +927,17 @@ xenFormatXLOS(virConfPtr conf, virDomainDefPtr def) if (xenConfigSetString(conf, "boot", boot) < 0) return -1; + if (def->cpu && def->cpu->nfeatures) { + for (i = 0; i < def->cpu->nfeatures; i++) { + if (def->cpu->features[i].policy == VIR_CPU_FEATURE_REQUIRE && + (STREQ(def->cpu->features[i].name, "vmx") || + STREQ(def->cpu->features[i].name, "svm"))) + if (xenConfigSetInt(conf, "nestedhvm", 1) < 0) + return -1; + } + } + + /* XXX floppy disks */ } else { if (def->os.bootloader && -- 2.9.3
From: Wim ten Have <wim.ten.have@oracle.com> Configurations containing (within its CPU element) following: Intel virtualization (VT-x): <feature policy='require' name='vmx'/> AMD virtualization (AMD-V): <feature policy='require' name='svm'/> generate nestedhvm=1 in/from xl format. Signed-off-by: Wim ten Have <wim.ten.have@oracle.com> --- tests/xlconfigdata/test-fullvirt-nestedhvm.cfg | 26 +++++++++++ tests/xlconfigdata/test-fullvirt-nestedhvm.xml | 61 ++++++++++++++++++++++++++ tests/xlconfigtest.c | 1 + 3 files changed, 88 insertions(+) create mode 100644 tests/xlconfigdata/test-fullvirt-nestedhvm.cfg create mode 100644 tests/xlconfigdata/test-fullvirt-nestedhvm.xml diff --git a/tests/xlconfigdata/test-fullvirt-nestedhvm.cfg b/tests/xlconfigdata/test-fullvirt-nestedhvm.cfg new file mode 100644 index 0000000..281f126 --- /dev/null +++ b/tests/xlconfigdata/test-fullvirt-nestedhvm.cfg @@ -0,0 +1,26 @@ +name = "XenGuest2" +uuid = "c7a5fdb2-cdaf-9455-926a-d65c16db1809" +maxmem = 579 +memory = 394 +vcpus = 1 +pae = 1 +acpi = 1 +apic = 1 +viridian = 0 +rtc_timeoffset = 0 +localtime = 0 +on_poweroff = "destroy" +on_reboot = "restart" +on_crash = "restart" +device_model = "/usr/lib/xen/bin/qemu-system-i386" +sdl = 0 +vnc = 1 +vncunused = 1 +vnclisten = "127.0.0.1" +vif = [ "mac=00:16:3e:66:92:9c,bridge=xenbr1,script=vif-bridge,model=e1000" ] +parallel = "none" +serial = "none" +builder = "hvm" +boot = "d" +nestedhvm = 1 +disk = [ "format=raw,vdev=hda,access=rw,backendtype=phy,target=/dev/HostVG/XenGuest2", "format=qcow2,vdev=hdb,access=rw,backendtype=qdisk,target=/var/lib/libvirt/images/XenGuest2-home", "format=raw,vdev=hdc,access=ro,backendtype=qdisk,devtype=cdrom,target=/root/boot.iso" ] diff --git a/tests/xlconfigdata/test-fullvirt-nestedhvm.xml b/tests/xlconfigdata/test-fullvirt-nestedhvm.xml new file mode 100644 index 0000000..a1f637a --- /dev/null +++ b/tests/xlconfigdata/test-fullvirt-nestedhvm.xml @@ -0,0 +1,61 @@ +<domain type='xen'> + <name>XenGuest2</name> + <uuid>c7a5fdb2-cdaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>592896</memory> + <currentMemory unit='KiB'>403456</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='x86_64' machine='xenfv'>hvm</type> + <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader> + <boot dev='cdrom'/> + </os> + <features> + <acpi/> + <apic/> + <pae/> + </features> + <cpu mode='host-passthrough'> + <feature policy='require' name='vmx'/> + </cpu> + <clock offset='variable' adjustment='0' basis='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>restart</on_crash> + <devices> + <emulator>/usr/lib/xen/bin/qemu-system-i386</emulator> + <disk type='block' device='disk'> + <driver name='phy' type='raw'/> + <source dev='/dev/HostVG/XenGuest2'/> + <target dev='hda' bus='ide'/> + <address type='drive' controller='0' bus='0' target='0' unit='0'/> + </disk> + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='/var/lib/libvirt/images/XenGuest2-home'/> + <target dev='hdb' bus='ide'/> + <address type='drive' controller='0' bus='0' target='0' unit='1'/> + </disk> + <disk type='file' device='cdrom'> + <driver name='qemu' type='raw'/> + <source file='/root/boot.iso'/> + <target dev='hdc' bus='ide'/> + <readonly/> + <address type='drive' controller='0' bus='1' target='0' unit='0'/> + </disk> + <controller type='ide' index='0'/> + <interface type='bridge'> + <mac address='00:16:3e:66:92:9c'/> + <source bridge='xenbr1'/> + <script path='vif-bridge'/> + <model type='e1000'/> + </interface> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'> + <listen type='address' address='127.0.0.1'/> + </graphics> + <video> + <model type='cirrus' vram='8192' heads='1' primary='yes'/> + </video> + </devices> +</domain> diff --git a/tests/xlconfigtest.c b/tests/xlconfigtest.c index e74e4d6..d2cc950 100644 --- a/tests/xlconfigtest.c +++ b/tests/xlconfigtest.c @@ -268,6 +268,7 @@ mymain(void) DO_TEST("fullvirt-hpet-timer"); DO_TEST("fullvirt-tsc-timer"); DO_TEST("fullvirt-multi-timer"); + DO_TEST("fullvirt-nestedhvm"); DO_TEST("paravirt-cmdline"); DO_TEST_FORMAT("paravirt-cmdline-extra-root", false); -- 2.9.3
On Wed, Mar 08, 2017 at 03:07:12PM +0100, Wim Ten Have wrote:
From: Wim ten Have <wim.ten.have@oracle.com>
This patch enhances host-passthrough capability to advertise the required vendor CPU virtualization feature which will be used to enable 'nestedhvm' in the libxl driver.
For Intel virtualization (VT-x) .. <cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu>
For AMD virtualization (AMD-V) .. <cpu mode='host-passthrough'> <feature policy='require' name='svm'/> </cpu>
If using host-passthrough or host-model, and the host's CPU includes vmx or svm, then I would expecte nested-HVM to be enabled, without needing any extra <feature> flag in the XML. That would match the semantics used with the QEMU driver. The only time we would need to use <feature> is if using mode=custom along with a named CPU model which lacks vmx/svm flags. BTW, I wonder how hard it would be to wire up the libxl driver to use the CPU model infrastructure in src/cpu ? It feels a little odd to use XML <cpu mode='host-passthrough'/> if we're not then making sure it actually uses host-passthrough when configuring the guest. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 03/08/2017 02:13 PM, Daniel P. Berrange wrote:
On Wed, Mar 08, 2017 at 03:07:12PM +0100, Wim Ten Have wrote:
From: Wim ten Have <wim.ten.have@oracle.com>
This patch enhances host-passthrough capability to advertise the required vendor CPU virtualization feature which will be used to enable 'nestedhvm' in the libxl driver.
For Intel virtualization (VT-x) .. <cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu>
For AMD virtualization (AMD-V) .. <cpu mode='host-passthrough'> <feature policy='require' name='svm'/> </cpu>
If using host-passthrough or host-model, and the host's CPU includes vmx or svm, then I would expecte nested-HVM to be enabled, without needing any extra <feature> flag in the XML. That would match the semantics used with the QEMU driver.
The only time we would need to use <feature> is if using mode=custom along with a named CPU model which lacks vmx/svm flags. Ah OK - I was kinda off unclear on that. So using host-passthrough should be enough then. (while making sure libxl checks if host->cpu does have vmx or svm in its features)
BTW, I wonder how hard it would be to wire up the libxl driver to use the CPU model infrastructure in src/cpu ? It feels a little odd to use XML <cpu mode='host-passthrough'/> if we're not then making sure it actually uses host-passthrough when configuring the guest. While xen libxl do allow to mangle the cpuid, it is meant for disabling features at this point. libxl follows a "host-model" first, meaning the default is to expose as much as features as possible to the guest (depending on whether it's PV or HVM). However, nested virt is a slightly special case since the toolstack will do more than simply unmasking vmx/svm bits (actually within libxl, a sysctl to Xen is performed to enable nested virt to the domain, in which case libxc will handle any vendor specific bits). IOW, even when we improve libxl cpuid policy management to the point we can wire up to libvirt cpu model infrastruture, we would still need to handle the nestedhvm special case (which btw this field would work even for ARM whenever supported).
Joao
On Wed, Mar 08, 2017 at 03:09:11PM +0000, Joao Martins wrote:
On 03/08/2017 02:13 PM, Daniel P. Berrange wrote:
On Wed, Mar 08, 2017 at 03:07:12PM +0100, Wim Ten Have wrote:
From: Wim ten Have <wim.ten.have@oracle.com>
This patch enhances host-passthrough capability to advertise the required vendor CPU virtualization feature which will be used to enable 'nestedhvm' in the libxl driver.
For Intel virtualization (VT-x) .. <cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu>
For AMD virtualization (AMD-V) .. <cpu mode='host-passthrough'> <feature policy='require' name='svm'/> </cpu>
If using host-passthrough or host-model, and the host's CPU includes vmx or svm, then I would expecte nested-HVM to be enabled, without needing any extra <feature> flag in the XML. That would match the semantics used with the QEMU driver.
The only time we would need to use <feature> is if using mode=custom along with a named CPU model which lacks vmx/svm flags. Ah OK - I was kinda off unclear on that. So using host-passthrough should be enough then. (while making sure libxl checks if host->cpu does have vmx or svm in its features)
BTW, I wonder how hard it would be to wire up the libxl driver to use the CPU model infrastructure in src/cpu ? It feels a little odd to use XML <cpu mode='host-passthrough'/> if we're not then making sure it actually uses host-passthrough when configuring the guest. While xen libxl do allow to mangle the cpuid, it is meant for disabling features at this point. libxl follows a "host-model" first, meaning the default is to expose as much as features as possible to the guest (depending on whether it's PV or HVM). However, nested virt is a slightly special case since the toolstack will do more than simply unmasking vmx/svm bits (actually within libxl, a sysctl to Xen is performed to enable nested virt to the domain, in which case libxc will handle any vendor specific bits). IOW, even when we improve libxl cpuid policy management to the point we can wire up to libvirt cpu model infrastruture, we would still need to handle the nestedhvm special case (which btw this field would work even for ARM whenever supported).
Yep, I see. So with libxl using host-model by default, then - If the host CPU includes svm/vmx, turn on nested virt in the guest - Allow use of <feature name="vmx" policy="disable"/> to block nested virt in the guest, even if available in host CPU Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
On 03/08/2017 03:07 PM, Daniel P. Berrange wrote:
On Wed, Mar 08, 2017 at 03:09:11PM +0000, Joao Martins wrote:
On 03/08/2017 02:13 PM, Daniel P. Berrange wrote:
On Wed, Mar 08, 2017 at 03:07:12PM +0100, Wim Ten Have wrote:
From: Wim ten Have <wim.ten.have@oracle.com>
This patch enhances host-passthrough capability to advertise the required vendor CPU virtualization feature which will be used to enable 'nestedhvm' in the libxl driver.
For Intel virtualization (VT-x) .. <cpu mode='host-passthrough'> <feature policy='require' name='vmx'/> </cpu>
For AMD virtualization (AMD-V) .. <cpu mode='host-passthrough'> <feature policy='require' name='svm'/> </cpu>
If using host-passthrough or host-model, and the host's CPU includes vmx or svm, then I would expecte nested-HVM to be enabled, without needing any extra <feature> flag in the XML. That would match the semantics used with the QEMU driver.
The only time we would need to use <feature> is if using mode=custom along with a named CPU model which lacks vmx/svm flags. Ah OK - I was kinda off unclear on that. So using host-passthrough should be enough then. (while making sure libxl checks if host->cpu does have vmx or svm in its features)
BTW, I wonder how hard it would be to wire up the libxl driver to use the CPU model infrastructure in src/cpu ? It feels a little odd to use XML <cpu mode='host-passthrough'/> if we're not then making sure it actually uses host-passthrough when configuring the guest. While xen libxl do allow to mangle the cpuid, it is meant for disabling features at this point. libxl follows a "host-model" first, meaning the default is to expose as much as features as possible to the guest (depending on whether it's PV or HVM). However, nested virt is a slightly special case since the toolstack will do more than simply unmasking vmx/svm bits (actually within libxl, a sysctl to Xen is performed to enable nested virt to the domain, in which case libxc will handle any vendor specific bits). IOW, even when we improve libxl cpuid policy management to the point we can wire up to libvirt cpu model infrastruture, we would still need to handle the nestedhvm special case (which btw this field would work even for ARM whenever supported).
Yep, I see. So with libxl using host-model by default, then
- If the host CPU includes svm/vmx, turn on nested virt in the guest - Allow use of <feature name="vmx" policy="disable"/> to block nested virt in the guest, even if available in host CPU
Cool, thanks! Joao
participants (3)
-
Daniel P. Berrange -
Joao Martins -
Wim Ten Have