[libvirt PATCH v3 0/3] Add support for vDPA network devices

vDPA network devices allow high-performance networking in a virtual machine by providing a wire-speed data path. These devices require a vendor-specific host driver but the data path follows the virtio specification. The support for vDPA devices was recently added to qemu. This allows libvirt to support these devices. This patchset requires that the device is configured on the host with the appropriate vendor-specific driver. This will create a chardev on the host at e.g. /dev/vhost-vdpa-0. That chardev path can then be used to define a new interface with type='vdpa'. Changes in v3: - rebased to latest master - various small fixes per comments on last review - investigated hotplug -- I have a preliminary libvirt patch, but it doesn't appear that hotplug is fully supported in qemu yet. - qemu never closes the fd when the device is removed, so the device cannot be reassigned to a different domain. - I decided not to post the preliminary hotplug patch until I can get more clarity from qemu developers. Jonathon Jongsma (3): conf: Add support for vDPA network devices qemu: add vhost-vdpa capability qemu: add vdpa support docs/formatdomain.rst | 24 ++++++++++++ docs/schemas/domaincommon.rng | 15 ++++++++ src/conf/domain_conf.c | 31 ++++++++++++++++ src/conf/domain_conf.h | 4 ++ src/conf/netdev_bandwidth_conf.c | 1 + src/libxl/libxl_conf.c | 1 + src/libxl/xen_common.c | 1 + src/lxc/lxc_controller.c | 1 + src/lxc/lxc_driver.c | 3 ++ src/lxc/lxc_process.c | 1 + src/qemu/qemu_capabilities.c | 4 ++ src/qemu/qemu_capabilities.h | 3 ++ src/qemu/qemu_command.c | 31 +++++++++++++++- src/qemu/qemu_command.h | 3 +- src/qemu/qemu_domain.c | 6 ++- src/qemu/qemu_hotplug.c | 15 +++++--- src/qemu/qemu_interface.c | 25 +++++++++++++ src/qemu/qemu_interface.h | 2 + src/qemu/qemu_migration.c | 10 ++++- src/qemu/qemu_process.c | 2 + src/qemu/qemu_validate.c | 15 ++++++++ src/vmx/vmx.c | 1 + .../caps_5.1.0.x86_64.xml | 1 + .../caps_5.2.0.x86_64.xml | 1 + .../net-vdpa.x86_64-latest.args | 37 +++++++++++++++++++ tests/qemuxml2argvdata/net-vdpa.xml | 28 ++++++++++++++ tests/qemuxml2argvmock.c | 11 +++++- tests/qemuxml2argvtest.c | 1 + tests/qemuxml2xmloutdata/net-vdpa.xml | 34 +++++++++++++++++ tests/qemuxml2xmltest.c | 1 + tools/virsh-domain.c | 1 + 31 files changed, 303 insertions(+), 11 deletions(-) create mode 100644 tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/net-vdpa.xml create mode 100644 tests/qemuxml2xmloutdata/net-vdpa.xml -- 2.26.2

This patch adds new schema and adds support for parsing and formatting domain configurations that include vdpa devices. vDPA network devices allow high-performance networking in a virtual machine by providing a wire-speed data path. These devices require a vendor-specific host driver but the data path follows the virtio specification. When a device on the host is bound to an appropriate vendor-specific driver, it will create a chardev on the host at e.g. /dev/vhost-vdpa-0. That chardev path can then be used to define a new interface with type='vdpa'. Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- docs/formatdomain.rst | 24 ++++++++++++++++++++++++ docs/schemas/domaincommon.rng | 15 +++++++++++++++ src/conf/domain_conf.c | 31 +++++++++++++++++++++++++++++++ src/conf/domain_conf.h | 4 ++++ src/conf/netdev_bandwidth_conf.c | 1 + src/libxl/libxl_conf.c | 1 + src/libxl/xen_common.c | 1 + src/lxc/lxc_controller.c | 1 + src/lxc/lxc_driver.c | 3 +++ src/lxc/lxc_process.c | 1 + src/qemu/qemu_command.c | 3 +++ src/qemu/qemu_domain.c | 4 +++- src/qemu/qemu_hotplug.c | 3 +++ src/qemu/qemu_interface.c | 2 ++ src/qemu/qemu_process.c | 2 ++ src/qemu/qemu_validate.c | 1 + src/vmx/vmx.c | 1 + tools/virsh-domain.c | 1 + 18 files changed, 98 insertions(+), 1 deletion(-) diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index 18e237c157..b0b38d8cf5 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst @@ -4643,6 +4643,30 @@ or stopping the guest. </devices> ... +:anchor:`<a id="elementsNICSVDPA"/>` + +vDPA devices +^^^^^^^^^^^^ + +A vDPA network device can be used to provide wire speed network performance +within a domain. A vDPA device is a specialized type of network device that +uses a datapath that complies with the virtio specification but has a +vendor-specific control path. To use such a device with libvirt, the host +device must already be bound to the appropriate device-specific vDPA driver. +This creates a vDPA char device (e.g. /dev/vhost-vdpa-0) that can be used to +assign the device to a libvirt domain. :since:`Since 6.8.0 (QEMU only, +requires QEMU 5.1.0 or newer)` + +:: + + ... + <devices> + <interface type='vdpa'> + <source dev='/dev/vhost-vdpa-0'/> + </interface> + </devices> + ... + :anchor:`<a id="elementsTeaming"/>` Teaming a virtio/hostdev NIC pair diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng index a1d6d19e2f..4708e3550f 100644 --- a/docs/schemas/domaincommon.rng +++ b/docs/schemas/domaincommon.rng @@ -3111,6 +3111,21 @@ <ref name="interface-options"/> </interleave> </group> + + <group> + <attribute name="type"> + <value>vdpa</value> + </attribute> + <interleave> + <element name="source"> + <attribute name="dev"> + <ref name="deviceName"/> + </attribute> + </element> + <ref name="interface-options"/> + </interleave> + </group> + </choice> <optional> <attribute name="trustGuestRxFilters"> diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 72ac4f4191..cb3f6ae4c3 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -550,6 +550,7 @@ VIR_ENUM_IMPL(virDomainNet, "direct", "hostdev", "udp", + "vdpa", ); VIR_ENUM_IMPL(virDomainNetModel, @@ -2502,6 +2503,10 @@ virDomainNetDefClear(virDomainNetDefPtr def) def->data.vhostuser = NULL; break; + case VIR_DOMAIN_NET_TYPE_VDPA: + VIR_FREE(def->data.vdpa.devicepath); + break; + case VIR_DOMAIN_NET_TYPE_SERVER: case VIR_DOMAIN_NET_TYPE_CLIENT: case VIR_DOMAIN_NET_TYPE_MCAST: @@ -12129,6 +12134,10 @@ virDomainNetDefParseXML(virDomainXMLOptionPtr xmlopt, if (virDomainChrSourceReconnectDefParseXML(&reconnect, cur, ctxt) < 0) goto error; + } else if (!dev + && def->type == VIR_DOMAIN_NET_TYPE_VDPA + && virXMLNodeNameEqual(cur, "source")) { + dev = virXMLPropString(cur, "dev"); } else if (!def->virtPortProfile && virXMLNodeNameEqual(cur, "virtualport")) { if (def->type == VIR_DOMAIN_NET_TYPE_NETWORK) { @@ -12386,6 +12395,16 @@ virDomainNetDefParseXML(virDomainXMLOptionPtr xmlopt, } break; + case VIR_DOMAIN_NET_TYPE_VDPA: + if (dev == NULL) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("No <source> 'dev' attribute " + "specified with <interface type='vdpa'/>")); + goto error; + } + def->data.vdpa.devicepath = g_steal_pointer(&dev); + break; + case VIR_DOMAIN_NET_TYPE_BRIDGE: if (bridge == NULL) { virReportError(VIR_ERR_INTERNAL_ERROR, "%s", @@ -12775,6 +12794,7 @@ virDomainNetDefParseXML(virDomainXMLOptionPtr xmlopt, case VIR_DOMAIN_NET_TYPE_DIRECT: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: break; case VIR_DOMAIN_NET_TYPE_LAST: default: @@ -26881,6 +26901,14 @@ virDomainNetDefFormat(virBufferPtr buf, } break; + case VIR_DOMAIN_NET_TYPE_VDPA: + if (def->data.vdpa.devicepath) { + virBufferEscapeString(buf, "<source dev='%s'", + def->data.vdpa.devicepath); + sourceLines++; + } + break; + case VIR_DOMAIN_NET_TYPE_USER: case VIR_DOMAIN_NET_TYPE_LAST: break; @@ -31094,6 +31122,7 @@ virDomainNetGetActualVirtPortProfile(const virDomainNetDef *iface) case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: default: return NULL; @@ -31926,6 +31955,7 @@ virDomainNetTypeSharesHostView(const virDomainNetDef *net) case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: break; } @@ -32190,6 +32220,7 @@ virDomainNetDefActualToNetworkPort(virDomainDefPtr dom, case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_USER: case VIR_DOMAIN_NET_TYPE_VHOSTUSER: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Unexpected network port type %s"), virDomainNetTypeToString(virDomainNetGetActualType(iface))); diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 14a376350c..1a1caad96d 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -875,6 +875,7 @@ typedef enum { VIR_DOMAIN_NET_TYPE_DIRECT, VIR_DOMAIN_NET_TYPE_HOSTDEV, VIR_DOMAIN_NET_TYPE_UDP, + VIR_DOMAIN_NET_TYPE_VDPA, VIR_DOMAIN_NET_TYPE_LAST } virDomainNetType; @@ -1048,6 +1049,9 @@ struct _virDomainNetDef { */ virDomainActualNetDefPtr actual; } network; + struct { + char *devicepath; + } vdpa; struct { char *brname; } bridge; diff --git a/src/conf/netdev_bandwidth_conf.c b/src/conf/netdev_bandwidth_conf.c index 396ac62019..4eb12e2951 100644 --- a/src/conf/netdev_bandwidth_conf.c +++ b/src/conf/netdev_bandwidth_conf.c @@ -315,6 +315,7 @@ bool virNetDevSupportsBandwidth(virDomainNetType type) case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: break; } diff --git a/src/libxl/libxl_conf.c b/src/libxl/libxl_conf.c index befd5eb965..23f1cb213e 100644 --- a/src/libxl/libxl_conf.c +++ b/src/libxl/libxl_conf.c @@ -1378,6 +1378,7 @@ libxlMakeNic(virDomainDefPtr def, case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_DIRECT: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("unsupported interface type %s"), diff --git a/src/libxl/xen_common.c b/src/libxl/xen_common.c index 56702a2a76..183b09671a 100644 --- a/src/libxl/xen_common.c +++ b/src/libxl/xen_common.c @@ -1792,6 +1792,7 @@ xenFormatNet(virConnectPtr conn, case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_USER: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Unsupported net type '%s'"), virDomainNetTypeToString(net->type)); return -1; diff --git a/src/lxc/lxc_controller.c b/src/lxc/lxc_controller.c index 37a28ac2f3..509faecdae 100644 --- a/src/lxc/lxc_controller.c +++ b/src/lxc/lxc_controller.c @@ -422,6 +422,7 @@ static int virLXCControllerGetNICIndexes(virLXCControllerPtr ctrl) case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Unsupported net type %s"), virDomainNetTypeToString(actualType)); diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index a530488dd2..4d3c5d9f63 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -3504,6 +3504,7 @@ lxcDomainAttachDeviceNetLive(virLXCDriverPtr driver, case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Network device type is not supported")); goto cleanup; @@ -3558,6 +3559,7 @@ lxcDomainAttachDeviceNetLive(virLXCDriverPtr driver, case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: default: /* no-op */ @@ -3999,6 +4001,7 @@ lxcDomainDetachDeviceNetLive(virDomainObjPtr vm, case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Only bridged veth devices can be detached")); goto cleanup; diff --git a/src/lxc/lxc_process.c b/src/lxc/lxc_process.c index 16969dbf33..d103ec6666 100644 --- a/src/lxc/lxc_process.c +++ b/src/lxc/lxc_process.c @@ -606,6 +606,7 @@ virLXCProcessSetupInterfaces(virLXCDriverPtr driver, case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_LAST: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_INTERNAL_ERROR, _("Unsupported network type %s"), virDomainNetTypeToString(type)); diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index bd98b0a97c..7b7176eb72 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -3691,6 +3691,7 @@ qemuBuildHostNetStr(virDomainNetDefPtr net, return NULL; break; + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_HOSTDEV: /* Should have been handled earlier via PCI/USB hotplug code. */ case VIR_DOMAIN_NET_TYPE_LAST: @@ -8107,6 +8108,7 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* nada */ break; @@ -8143,6 +8145,7 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* These types don't use a network device on the host, but * instead use some other type of connection to the emulated diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 0ed132a829..82a6a934e1 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -5184,7 +5184,8 @@ qemuDomainDeviceNetDefPostParse(virDomainNetDefPtr net, const virDomainDef *def, virQEMUCapsPtr qemuCaps) { - if (net->type != VIR_DOMAIN_NET_TYPE_HOSTDEV && + if (net->type != VIR_DOMAIN_NET_TYPE_VDPA && + net->type != VIR_DOMAIN_NET_TYPE_HOSTDEV && !virDomainNetGetModelString(net) && virDomainNetResolveActualType(net) != VIR_DOMAIN_NET_TYPE_HOSTDEV) net->model = qemuDomainDefaultNetModel(def, qemuCaps); @@ -9318,6 +9319,7 @@ qemuDomainNetSupportsMTU(virDomainNetType type) case VIR_DOMAIN_NET_TYPE_DIRECT: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: break; } diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index e2c6e14c2e..78dd5e9f19 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -1340,6 +1340,7 @@ qemuDomainAttachNetDevice(virQEMUDriverPtr driver, case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("hotplug of interface type of %s is not implemented yet"), @@ -3389,6 +3390,7 @@ qemuDomainChangeNetFilter(virDomainObjPtr vm, case VIR_DOMAIN_NET_TYPE_DIRECT: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("filters not supported on interfaces of type %s"), virDomainNetTypeToString(virDomainNetGetActualType(newdev))); @@ -3726,6 +3728,7 @@ qemuDomainChangeNet(virQEMUDriverPtr driver, case VIR_DOMAIN_NET_TYPE_VHOSTUSER: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("unable to change config on '%s' network type"), virDomainNetTypeToString(newdev->type)); diff --git a/src/qemu/qemu_interface.c b/src/qemu/qemu_interface.c index cbf3d99981..b24f9060a9 100644 --- a/src/qemu/qemu_interface.c +++ b/src/qemu/qemu_interface.c @@ -118,6 +118,7 @@ qemuInterfaceStartDevice(virDomainNetDefPtr net) case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* these types all require no action */ break; @@ -203,6 +204,7 @@ qemuInterfaceStopDevice(virDomainNetDefPtr net) case VIR_DOMAIN_NET_TYPE_UDP: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* these types all require no action */ break; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index dd60fb0ddf..6a7f87fb0e 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3363,6 +3363,7 @@ qemuProcessNotifyNets(virDomainDefPtr def) case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: break; } @@ -7570,6 +7571,7 @@ void qemuProcessStop(virQEMUDriverPtr driver, case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* No special cleanup procedure for these types. */ break; diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index 070f1c962b..25f7866e5c 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -1145,6 +1145,7 @@ qemuValidateNetSupportsCoalesce(virDomainNetType type) case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: break; } diff --git a/src/vmx/vmx.c b/src/vmx/vmx.c index 4b1b04c6e1..6e0fd61f60 100644 --- a/src/vmx/vmx.c +++ b/src/vmx/vmx.c @@ -3833,6 +3833,7 @@ virVMXFormatEthernet(virDomainNetDefPtr def, int controller, case VIR_DOMAIN_NET_TYPE_DIRECT: case VIR_DOMAIN_NET_TYPE_HOSTDEV: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Unsupported net type '%s'"), virDomainNetTypeToString(def->type)); return -1; diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 08496cc5bf..ba31154f2f 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -1006,6 +1006,7 @@ cmdAttachInterface(vshControl *ctl, const vshCmd *cmd) case VIR_DOMAIN_NET_TYPE_CLIENT: case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_UDP: + case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_LAST: vshError(ctl, _("No support for %s in command 'attach-interface'"), -- 2.26.2

Recent versions of qemu added the -netdev vhost-vdpa device. This capability allows libvirt to know whether this is supported. Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- src/qemu/qemu_capabilities.c | 4 ++++ src/qemu/qemu_capabilities.h | 3 +++ tests/qemucapabilitiesdata/caps_5.1.0.x86_64.xml | 1 + tests/qemucapabilitiesdata/caps_5.2.0.x86_64.xml | 1 + 4 files changed, 9 insertions(+) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index 2cc0c61588..b19928f68d 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -597,6 +597,9 @@ VIR_ENUM_IMPL(virQEMUCaps, "spapr-tpm-proxy", "numa.hmat", "blockdev-hostdev-scsi", + + /* 380 */ + "netdev.vhost-vdpa", ); @@ -1526,6 +1529,7 @@ static struct virQEMUCapsStringFlags virQEMUCapsQMPSchemaQueries[] = { { "migrate-set-parameters/arg-type/downtime-limit", QEMU_CAPS_MIGRATION_PARAM_DOWNTIME }, { "migrate-set-parameters/arg-type/xbzrle-cache-size", QEMU_CAPS_MIGRATION_PARAM_XBZRLE_CACHE_SIZE }, { "set-numa-node/arg-type/+hmat-lb", QEMU_CAPS_NUMA_HMAT }, + { "netdev_add/arg-type/+vhost-vdpa", QEMU_CAPS_NETDEV_VHOST_VDPA }, }; typedef struct _virQEMUCapsObjectTypeProps virQEMUCapsObjectTypeProps; diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index 5d08941538..b6110f1c34 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -578,6 +578,9 @@ typedef enum { /* virQEMUCapsFlags grouping marker for syntax-check */ QEMU_CAPS_NUMA_HMAT, /* -numa hmat */ QEMU_CAPS_BLOCKDEV_HOSTDEV_SCSI, /* -blockdev used for (i)SCSI hostdevs */ + /* 380 */ + QEMU_CAPS_NETDEV_VHOST_VDPA, /* -netdev vhost-vdpa*/ + QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_5.1.0.x86_64.xml b/tests/qemucapabilitiesdata/caps_5.1.0.x86_64.xml index 7496ff1379..0fd2f3b816 100644 --- a/tests/qemucapabilitiesdata/caps_5.1.0.x86_64.xml +++ b/tests/qemucapabilitiesdata/caps_5.1.0.x86_64.xml @@ -242,6 +242,7 @@ <flag name='intel-iommu.aw-bits'/> <flag name='numa.hmat'/> <flag name='blockdev-hostdev-scsi'/> + <flag name='netdev.vhost-vdpa'/> <version>5001000</version> <kvmVersion>0</kvmVersion> <microcodeVersion>43100242</microcodeVersion> diff --git a/tests/qemucapabilitiesdata/caps_5.2.0.x86_64.xml b/tests/qemucapabilitiesdata/caps_5.2.0.x86_64.xml index 151bd18137..e4686000a9 100644 --- a/tests/qemucapabilitiesdata/caps_5.2.0.x86_64.xml +++ b/tests/qemucapabilitiesdata/caps_5.2.0.x86_64.xml @@ -242,6 +242,7 @@ <flag name='intel-iommu.aw-bits'/> <flag name='numa.hmat'/> <flag name='blockdev-hostdev-scsi'/> + <flag name='netdev.vhost-vdpa'/> <version>5001050</version> <kvmVersion>0</kvmVersion> <microcodeVersion>43100243</microcodeVersion> -- 2.26.2

Enable <interface type='vdpa'> for qemu domains. This provides basic support and does not support hotplug or migration. Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- src/qemu/qemu_command.c | 30 +++++++++++++-- src/qemu/qemu_command.h | 3 +- src/qemu/qemu_domain.c | 6 ++- src/qemu/qemu_hotplug.c | 12 +++--- src/qemu/qemu_interface.c | 23 ++++++++++++ src/qemu/qemu_interface.h | 2 + src/qemu/qemu_migration.c | 10 ++++- src/qemu/qemu_validate.c | 14 +++++++ .../net-vdpa.x86_64-latest.args | 37 +++++++++++++++++++ tests/qemuxml2argvdata/net-vdpa.xml | 28 ++++++++++++++ tests/qemuxml2argvmock.c | 11 +++++- tests/qemuxml2argvtest.c | 1 + tests/qemuxml2xmloutdata/net-vdpa.xml | 34 +++++++++++++++++ tests/qemuxml2xmltest.c | 1 + 14 files changed, 199 insertions(+), 13 deletions(-) create mode 100644 tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/net-vdpa.xml create mode 100644 tests/qemuxml2xmloutdata/net-vdpa.xml diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 7b7176eb72..1c8c723d58 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -3553,7 +3553,8 @@ qemuBuildHostNetStr(virDomainNetDefPtr net, size_t tapfdSize, char **vhostfd, size_t vhostfdSize, - const char *slirpfd) + const char *slirpfd, + const char *vdpadev) { bool is_tap = false; virDomainNetType netType = virDomainNetGetActualType(net); @@ -3692,6 +3693,12 @@ qemuBuildHostNetStr(virDomainNetDefPtr net, break; case VIR_DOMAIN_NET_TYPE_VDPA: + /* Caller will pass the fd to qemu with add-fd */ + if (virJSONValueObjectCreate(&netprops, "s:type", "vhost-vdpa", NULL) < 0 || + virJSONValueObjectAppendString(netprops, "vhostdev", vdpadev) < 0) + return NULL; + break; + case VIR_DOMAIN_NET_TYPE_HOSTDEV: /* Should have been handled earlier via PCI/USB hotplug code. */ case VIR_DOMAIN_NET_TYPE_LAST: @@ -8017,6 +8024,8 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, char **tapfdName = NULL; char **vhostfdName = NULL; g_autofree char *slirpfdName = NULL; + g_autofree char *vdpafdName = NULL; + int vdpafd = -1; virDomainNetType actualType = virDomainNetGetActualType(net); const virNetDevBandwidth *actualBandwidth; bool requireNicdev = false; @@ -8102,13 +8111,17 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, break; + case VIR_DOMAIN_NET_TYPE_VDPA: + if ((vdpafd = qemuInterfaceVDPAConnect(net)) < 0) + goto cleanup; + break; + case VIR_DOMAIN_NET_TYPE_USER: case VIR_DOMAIN_NET_TYPE_SERVER: case VIR_DOMAIN_NET_TYPE_CLIENT: case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_UDP: - case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* nada */ break; @@ -8225,13 +8238,24 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, vhostfd[i] = -1; } + if (vdpafd > 0) { + g_autofree char *fdset = NULL; + + virCommandPassFD(cmd, vdpafd, VIR_COMMAND_PASS_FD_CLOSE_PARENT); + fdset = qemuVirCommandGetFDSet(cmd, vdpafd); + if (!fdset) + goto cleanup; + virCommandAddArgList(cmd, "-add-fd", fdset, NULL); + vdpafdName = qemuVirCommandGetDevSet(cmd, vdpafd); + } + if (chardev) virCommandAddArgList(cmd, "-chardev", chardev, NULL); if (!(hostnetprops = qemuBuildHostNetStr(net, tapfdName, tapfdSize, vhostfdName, vhostfdSize, - slirpfdName))) + slirpfdName, vdpafdName))) goto cleanup; if (!(host = virQEMUBuildNetdevCommandlineFromJSON(hostnetprops, diff --git a/src/qemu/qemu_command.h b/src/qemu/qemu_command.h index 89d99b111f..8db51f93b1 100644 --- a/src/qemu/qemu_command.h +++ b/src/qemu/qemu_command.h @@ -99,7 +99,8 @@ virJSONValuePtr qemuBuildHostNetStr(virDomainNetDefPtr net, size_t tapfdSize, char **vhostfd, size_t vhostfdSize, - const char *slirpfd); + const char *slirpfd, + const char *vdpadev); /* Current, best practice */ char *qemuBuildNicDevStr(virDomainDefPtr def, diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 82a6a934e1..2bacd2753c 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -5184,8 +5184,10 @@ qemuDomainDeviceNetDefPostParse(virDomainNetDefPtr net, const virDomainDef *def, virQEMUCapsPtr qemuCaps) { - if (net->type != VIR_DOMAIN_NET_TYPE_VDPA && - net->type != VIR_DOMAIN_NET_TYPE_HOSTDEV && + if (net->type == VIR_DOMAIN_NET_TYPE_VDPA && + !virDomainNetGetModelString(net)) + net->model = VIR_DOMAIN_NET_MODEL_VIRTIO; + else if (net->type != VIR_DOMAIN_NET_TYPE_HOSTDEV && !virDomainNetGetModelString(net) && virDomainNetResolveActualType(net) != VIR_DOMAIN_NET_TYPE_HOSTDEV) net->model = qemuDomainDefaultNetModel(def, qemuCaps); diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 78dd5e9f19..45c07a7f4e 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -1389,7 +1389,7 @@ qemuDomainAttachNetDevice(virQEMUDriverPtr driver, if (!(netprops = qemuBuildHostNetStr(net, tapfdName, tapfdSize, vhostfdName, vhostfdSize, - slirpfdName))) + slirpfdName, NULL))) goto cleanup; qemuDomainObjEnterMonitor(driver, vm); @@ -3484,8 +3484,9 @@ qemuDomainChangeNet(virQEMUDriverPtr driver, olddev = *devslot; oldType = virDomainNetGetActualType(olddev); - if (oldType == VIR_DOMAIN_NET_TYPE_HOSTDEV) { - /* no changes are possible to a type='hostdev' interface */ + if (oldType == VIR_DOMAIN_NET_TYPE_HOSTDEV || + oldType == VIR_DOMAIN_NET_TYPE_VDPA) { + /* no changes are possible to a type='hostdev' or type='vdpa' interface */ virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("cannot change config of '%s' network type"), virDomainNetTypeToString(oldType)); @@ -3672,8 +3673,9 @@ qemuDomainChangeNet(virQEMUDriverPtr driver, newType = virDomainNetGetActualType(newdev); - if (newType == VIR_DOMAIN_NET_TYPE_HOSTDEV) { - /* can't turn it into a type='hostdev' interface */ + if (newType == VIR_DOMAIN_NET_TYPE_HOSTDEV || + newType == VIR_DOMAIN_NET_TYPE_VDPA) { + /* can't turn it into a type='hostdev' or type='vdpa' interface */ virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("cannot change network interface type to '%s'"), virDomainNetTypeToString(newType)); diff --git a/src/qemu/qemu_interface.c b/src/qemu/qemu_interface.c index b24f9060a9..3714828fe1 100644 --- a/src/qemu/qemu_interface.c +++ b/src/qemu/qemu_interface.c @@ -638,6 +638,29 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def, } +/* qemuInterfaceVDPAConnect: + * @net: pointer to the VM's interface description + * + * returns: file descriptor of the vdpa device + * + * Called *only* called if actualType is VIR_DOMAIN_NET_TYPE_VDPA + */ +int +qemuInterfaceVDPAConnect(virDomainNetDefPtr net) +{ + int fd; + + if ((fd = open(net->data.vdpa.devicepath, O_RDWR)) < 0) { + virReportSystemError(errno, + _("Unable to open '%s' for vdpa device"), + net->data.vdpa.devicepath); + return -1; + } + + return fd; +} + + qemuSlirpPtr qemuInterfacePrepareSlirp(virQEMUDriverPtr driver, virDomainNetDefPtr net) diff --git a/src/qemu/qemu_interface.h b/src/qemu/qemu_interface.h index 3dcefc6a12..1ba24f0a6f 100644 --- a/src/qemu/qemu_interface.h +++ b/src/qemu/qemu_interface.h @@ -58,3 +58,5 @@ int qemuInterfaceOpenVhostNet(virDomainDefPtr def, qemuSlirpPtr qemuInterfacePrepareSlirp(virQEMUDriverPtr driver, virDomainNetDefPtr net); + +int qemuInterfaceVDPAConnect(virDomainNetDefPtr net) G_GNUC_NO_INLINE; diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a530c17582..d82de9455c 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1372,7 +1372,15 @@ qemuMigrationSrcIsAllowed(virQEMUDriverPtr driver, for (i = 0; i < vm->def->nnets; i++) { virDomainNetDefPtr net = vm->def->nets[i]; - qemuSlirpPtr slirp = QEMU_DOMAIN_NETWORK_PRIVATE(net)->slirp; + qemuSlirpPtr slirp; + + if (net->type == VIR_DOMAIN_NET_TYPE_VDPA) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("a vDPA device cannot be migrated")); + return false; + } + + slirp = QEMU_DOMAIN_NETWORK_PRIVATE(net)->slirp; if (slirp && !qemuSlirpHasFeature(slirp, QEMU_SLIRP_FEATURE_MIGRATE)) { virReportError(VIR_ERR_OPERATION_INVALID, "%s", diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index 25f7866e5c..719cd19115 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -1245,6 +1245,20 @@ qemuValidateDomainDeviceDefNetwork(const virDomainNetDef *net, } } } + } else if (net->type == VIR_DOMAIN_NET_TYPE_VDPA) { + if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_NETDEV_VHOST_VDPA)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("vDPA device is not supported with this QEMU binary")); + return -1; + } + + if (net->model != VIR_DOMAIN_NET_MODEL_VIRTIO) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("invalid model for interface of type '%s': '%s'"), + virDomainNetTypeToString(net->type), + virDomainNetModelTypeToString(net->model)); + return -1; + } } else if (net->guestIP.nroutes || net->guestIP.nips) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Invalid attempt to set network interface " diff --git a/tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args b/tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args new file mode 100644 index 0000000000..8e76ac7794 --- /dev/null +++ b/tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args @@ -0,0 +1,37 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest1/.config \ +QEMU_AUDIO_DRV=none \ +/usr/bin/qemu-system-i386 \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object secret,id=masterKey0,format=raw,\ +file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ +-machine pc,accel=tcg,usb=off,dump-guest-core=off \ +-cpu qemu64 \ +-m 214 \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server,nowait \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-no-acpi \ +-boot strict=on \ +-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ +-add-fd set=0,fd=1732 \ +-netdev vhost-vdpa,vhostdev=/dev/fdset/0,id=hostnet0 \ +-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:95:db:c0,bus=pci.0,\ +addr=0x2 \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,\ +resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxml2argvdata/net-vdpa.xml b/tests/qemuxml2argvdata/net-vdpa.xml new file mode 100644 index 0000000000..30cca7eb6e --- /dev/null +++ b/tests/qemuxml2argvdata/net-vdpa.xml @@ -0,0 +1,28 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-i386</emulator> + <controller type='usb' index='0'/> + <controller type='ide' index='0'/> + <controller type='pci' index='0' model='pci-root'/> + <interface type='vdpa'> + <mac address='52:54:00:95:db:c0'/> + <source dev='/dev/vhost-vdpa-0'/> + </interface> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2argvmock.c b/tests/qemuxml2argvmock.c index e5841bc8e3..516776697f 100644 --- a/tests/qemuxml2argvmock.c +++ b/tests/qemuxml2argvmock.c @@ -205,7 +205,7 @@ virHostGetDRMRenderNode(void) static void (*real_virCommandPassFD)(virCommandPtr cmd, int fd, unsigned int flags); -static const int testCommandPassSafeFDs[] = { 1730, 1731 }; +static const int testCommandPassSafeFDs[] = { 1730, 1731, 1732 }; void virCommandPassFD(virCommandPtr cmd, @@ -283,3 +283,12 @@ qemuBuildTPMOpenBackendFDs(const char *tpmdev G_GNUC_UNUSED, *cancelfd = 1731; return 0; } + + +int +qemuInterfaceVDPAConnect(virDomainNetDefPtr net G_GNUC_UNUSED) +{ + if (fcntl(1732, F_GETFD) != -1) + abort(); + return 1732; +} diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index e93948e3fc..000d6919f2 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -1446,6 +1446,7 @@ mymain(void) QEMU_CAPS_DEVICE_VFIO_PCI); DO_TEST_FAILURE("net-hostdev-fail", QEMU_CAPS_DEVICE_VFIO_PCI); + DO_TEST_CAPS_LATEST("net-vdpa"); DO_TEST("hostdev-pci-multifunction", QEMU_CAPS_KVM, diff --git a/tests/qemuxml2xmloutdata/net-vdpa.xml b/tests/qemuxml2xmloutdata/net-vdpa.xml new file mode 100644 index 0000000000..b362405c14 --- /dev/null +++ b/tests/qemuxml2xmloutdata/net-vdpa.xml @@ -0,0 +1,34 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-i386</emulator> + <controller type='usb' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='ide' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <interface type='vdpa'> + <mac address='52:54:00:95:db:c0'/> + <source dev='/dev/vhost-vdpa-0'/> + <model type='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </interface> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c index 39a9da874f..ac33e87698 100644 --- a/tests/qemuxml2xmltest.c +++ b/tests/qemuxml2xmltest.c @@ -497,6 +497,7 @@ mymain(void) DO_TEST("net-mtu", NONE); DO_TEST("net-coalesce", NONE); DO_TEST("net-many-models", NONE); + DO_TEST("net-vdpa", QEMU_CAPS_NETDEV_VHOST_VDPA); DO_TEST("serial-tcp-tlsx509-chardev", NONE); DO_TEST("serial-tcp-tlsx509-chardev-notls", NONE); -- 2.26.2

On 9/11/20 5:17 PM, Jonathon Jongsma wrote:
Enable <interface type='vdpa'> for qemu domains. This provides basic support and does not support hotplug or migration.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> ---
Just a couple nits about error message wording, but looks fine, assuming that it works with an actual device. Reviewed-by: Laine Stump <laine@redhat.com> (with the small error messages fixed and demonstrating that it works with a device)
src/qemu/qemu_command.c | 30 +++++++++++++-- src/qemu/qemu_command.h | 3 +- src/qemu/qemu_domain.c | 6 ++- src/qemu/qemu_hotplug.c | 12 +++--- src/qemu/qemu_interface.c | 23 ++++++++++++ src/qemu/qemu_interface.h | 2 + src/qemu/qemu_migration.c | 10 ++++- src/qemu/qemu_validate.c | 14 +++++++ .../net-vdpa.x86_64-latest.args | 37 +++++++++++++++++++ tests/qemuxml2argvdata/net-vdpa.xml | 28 ++++++++++++++ tests/qemuxml2argvmock.c | 11 +++++- tests/qemuxml2argvtest.c | 1 + tests/qemuxml2xmloutdata/net-vdpa.xml | 34 +++++++++++++++++ tests/qemuxml2xmltest.c | 1 + 14 files changed, 199 insertions(+), 13 deletions(-) create mode 100644 tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/net-vdpa.xml create mode 100644 tests/qemuxml2xmloutdata/net-vdpa.xml
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 7b7176eb72..1c8c723d58 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -3553,7 +3553,8 @@ qemuBuildHostNetStr(virDomainNetDefPtr net, size_t tapfdSize, char **vhostfd, size_t vhostfdSize, - const char *slirpfd) + const char *slirpfd, + const char *vdpadev) { bool is_tap = false; virDomainNetType netType = virDomainNetGetActualType(net); @@ -3692,6 +3693,12 @@ qemuBuildHostNetStr(virDomainNetDefPtr net, break;
case VIR_DOMAIN_NET_TYPE_VDPA: + /* Caller will pass the fd to qemu with add-fd */ + if (virJSONValueObjectCreate(&netprops, "s:type", "vhost-vdpa", NULL) < 0 || + virJSONValueObjectAppendString(netprops, "vhostdev", vdpadev) < 0) + return NULL; + break; + case VIR_DOMAIN_NET_TYPE_HOSTDEV: /* Should have been handled earlier via PCI/USB hotplug code. */ case VIR_DOMAIN_NET_TYPE_LAST: @@ -8017,6 +8024,8 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, char **tapfdName = NULL; char **vhostfdName = NULL; g_autofree char *slirpfdName = NULL; + g_autofree char *vdpafdName = NULL; + int vdpafd = -1; virDomainNetType actualType = virDomainNetGetActualType(net); const virNetDevBandwidth *actualBandwidth; bool requireNicdev = false; @@ -8102,13 +8111,17 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver,
break;
+ case VIR_DOMAIN_NET_TYPE_VDPA: + if ((vdpafd = qemuInterfaceVDPAConnect(net)) < 0) + goto cleanup; + break; + case VIR_DOMAIN_NET_TYPE_USER: case VIR_DOMAIN_NET_TYPE_SERVER: case VIR_DOMAIN_NET_TYPE_CLIENT: case VIR_DOMAIN_NET_TYPE_MCAST: case VIR_DOMAIN_NET_TYPE_INTERNAL: case VIR_DOMAIN_NET_TYPE_UDP: - case VIR_DOMAIN_NET_TYPE_VDPA: case VIR_DOMAIN_NET_TYPE_LAST: /* nada */ break; @@ -8225,13 +8238,24 @@ qemuBuildInterfaceCommandLine(virQEMUDriverPtr driver, vhostfd[i] = -1; }
+ if (vdpafd > 0) { + g_autofree char *fdset = NULL; + + virCommandPassFD(cmd, vdpafd, VIR_COMMAND_PASS_FD_CLOSE_PARENT); + fdset = qemuVirCommandGetFDSet(cmd, vdpafd); + if (!fdset) + goto cleanup; + virCommandAddArgList(cmd, "-add-fd", fdset, NULL); + vdpafdName = qemuVirCommandGetDevSet(cmd, vdpafd); + } + if (chardev) virCommandAddArgList(cmd, "-chardev", chardev, NULL);
if (!(hostnetprops = qemuBuildHostNetStr(net, tapfdName, tapfdSize, vhostfdName, vhostfdSize, - slirpfdName))) + slirpfdName, vdpafdName))) goto cleanup;
if (!(host = virQEMUBuildNetdevCommandlineFromJSON(hostnetprops, diff --git a/src/qemu/qemu_command.h b/src/qemu/qemu_command.h index 89d99b111f..8db51f93b1 100644 --- a/src/qemu/qemu_command.h +++ b/src/qemu/qemu_command.h @@ -99,7 +99,8 @@ virJSONValuePtr qemuBuildHostNetStr(virDomainNetDefPtr net, size_t tapfdSize, char **vhostfd, size_t vhostfdSize, - const char *slirpfd); + const char *slirpfd, + const char *vdpadev);
/* Current, best practice */ char *qemuBuildNicDevStr(virDomainDefPtr def, diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 82a6a934e1..2bacd2753c 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -5184,8 +5184,10 @@ qemuDomainDeviceNetDefPostParse(virDomainNetDefPtr net, const virDomainDef *def, virQEMUCapsPtr qemuCaps) { - if (net->type != VIR_DOMAIN_NET_TYPE_VDPA && - net->type != VIR_DOMAIN_NET_TYPE_HOSTDEV && + if (net->type == VIR_DOMAIN_NET_TYPE_VDPA && + !virDomainNetGetModelString(net)) + net->model = VIR_DOMAIN_NET_MODEL_VIRTIO; + else if (net->type != VIR_DOMAIN_NET_TYPE_HOSTDEV && !virDomainNetGetModelString(net) && virDomainNetResolveActualType(net) != VIR_DOMAIN_NET_TYPE_HOSTDEV) net->model = qemuDomainDefaultNetModel(def, qemuCaps); diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 78dd5e9f19..45c07a7f4e 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -1389,7 +1389,7 @@ qemuDomainAttachNetDevice(virQEMUDriverPtr driver, if (!(netprops = qemuBuildHostNetStr(net, tapfdName, tapfdSize, vhostfdName, vhostfdSize, - slirpfdName))) + slirpfdName, NULL))) goto cleanup;
qemuDomainObjEnterMonitor(driver, vm); @@ -3484,8 +3484,9 @@ qemuDomainChangeNet(virQEMUDriverPtr driver, olddev = *devslot;
oldType = virDomainNetGetActualType(olddev); - if (oldType == VIR_DOMAIN_NET_TYPE_HOSTDEV) { - /* no changes are possible to a type='hostdev' interface */ + if (oldType == VIR_DOMAIN_NET_TYPE_HOSTDEV || + oldType == VIR_DOMAIN_NET_TYPE_VDPA) { + /* no changes are possible to a type='hostdev' or type='vdpa' interface */ virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("cannot change config of '%s' network type"),
Officially not your problem, but this message should have said "network interface type"...
virDomainNetTypeToString(oldType)); @@ -3672,8 +3673,9 @@ qemuDomainChangeNet(virQEMUDriverPtr driver,
newType = virDomainNetGetActualType(newdev);
- if (newType == VIR_DOMAIN_NET_TYPE_HOSTDEV) { - /* can't turn it into a type='hostdev' interface */ + if (newType == VIR_DOMAIN_NET_TYPE_HOSTDEV || + newType == VIR_DOMAIN_NET_TYPE_VDPA) { + /* can't turn it into a type='hostdev' or type='vdpa' interface */ virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("cannot change network interface type to '%s'"), virDomainNetTypeToString(newType)); diff --git a/src/qemu/qemu_interface.c b/src/qemu/qemu_interface.c index b24f9060a9..3714828fe1 100644 --- a/src/qemu/qemu_interface.c +++ b/src/qemu/qemu_interface.c @@ -638,6 +638,29 @@ qemuInterfaceBridgeConnect(virDomainDefPtr def, }
+/* qemuInterfaceVDPAConnect: + * @net: pointer to the VM's interface description + * + * returns: file descriptor of the vdpa device + * + * Called *only* called if actualType is VIR_DOMAIN_NET_TYPE_VDPA + */ +int +qemuInterfaceVDPAConnect(virDomainNetDefPtr net) +{ + int fd; + + if ((fd = open(net->data.vdpa.devicepath, O_RDWR)) < 0) { + virReportSystemError(errno, + _("Unable to open '%s' for vdpa device"), + net->data.vdpa.devicepath); + return -1; + } + + return fd; +} + + qemuSlirpPtr qemuInterfacePrepareSlirp(virQEMUDriverPtr driver, virDomainNetDefPtr net) diff --git a/src/qemu/qemu_interface.h b/src/qemu/qemu_interface.h index 3dcefc6a12..1ba24f0a6f 100644 --- a/src/qemu/qemu_interface.h +++ b/src/qemu/qemu_interface.h @@ -58,3 +58,5 @@ int qemuInterfaceOpenVhostNet(virDomainDefPtr def,
qemuSlirpPtr qemuInterfacePrepareSlirp(virQEMUDriverPtr driver, virDomainNetDefPtr net); + +int qemuInterfaceVDPAConnect(virDomainNetDefPtr net) G_GNUC_NO_INLINE; diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a530c17582..d82de9455c 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1372,7 +1372,15 @@ qemuMigrationSrcIsAllowed(virQEMUDriverPtr driver,
for (i = 0; i < vm->def->nnets; i++) { virDomainNetDefPtr net = vm->def->nets[i]; - qemuSlirpPtr slirp = QEMU_DOMAIN_NETWORK_PRIVATE(net)->slirp; + qemuSlirpPtr slirp; + + if (net->type == VIR_DOMAIN_NET_TYPE_VDPA) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("a vDPA device cannot be migrated"));
s/a vDPA device/vDPA devices/ Also, I wonder if the capitalization in the error messages should match the (lack of) capitalization in the XML (no strong opinion)
+ return false; + } + + slirp = QEMU_DOMAIN_NETWORK_PRIVATE(net)->slirp;
if (slirp && !qemuSlirpHasFeature(slirp, QEMU_SLIRP_FEATURE_MIGRATE)) { virReportError(VIR_ERR_OPERATION_INVALID, "%s", diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index 25f7866e5c..719cd19115 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -1245,6 +1245,20 @@ qemuValidateDomainDeviceDefNetwork(const virDomainNetDef *net, } } } + } else if (net->type == VIR_DOMAIN_NET_TYPE_VDPA) { + if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_NETDEV_VHOST_VDPA)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("vDPA device is not supported with this QEMU binary"));
s/device is/devices are/
+ return -1; + } + + if (net->model != VIR_DOMAIN_NET_MODEL_VIRTIO) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("invalid model for interface of type '%s': '%s'"), + virDomainNetTypeToString(net->type), + virDomainNetModelTypeToString(net->model)); + return -1; + } } else if (net->guestIP.nroutes || net->guestIP.nips) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Invalid attempt to set network interface " diff --git a/tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args b/tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args new file mode 100644 index 0000000000..8e76ac7794 --- /dev/null +++ b/tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args @@ -0,0 +1,37 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-QEMUGuest1 \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest1/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest1/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest1/.config \ +QEMU_AUDIO_DRV=none \ +/usr/bin/qemu-system-i386 \ +-name guest=QEMUGuest1,debug-threads=on \ +-S \ +-object secret,id=masterKey0,format=raw,\ +file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ +-machine pc,accel=tcg,usb=off,dump-guest-core=off \ +-cpu qemu64 \ +-m 214 \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server,nowait \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-no-acpi \ +-boot strict=on \ +-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ +-add-fd set=0,fd=1732 \ +-netdev vhost-vdpa,vhostdev=/dev/fdset/0,id=hostnet0 \ +-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:95:db:c0,bus=pci.0,\ +addr=0x2 \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,\ +resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxml2argvdata/net-vdpa.xml b/tests/qemuxml2argvdata/net-vdpa.xml new file mode 100644 index 0000000000..30cca7eb6e --- /dev/null +++ b/tests/qemuxml2argvdata/net-vdpa.xml @@ -0,0 +1,28 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-i386</emulator> + <controller type='usb' index='0'/> + <controller type='ide' index='0'/> + <controller type='pci' index='0' model='pci-root'/> + <interface type='vdpa'> + <mac address='52:54:00:95:db:c0'/> + <source dev='/dev/vhost-vdpa-0'/> + </interface> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2argvmock.c b/tests/qemuxml2argvmock.c index e5841bc8e3..516776697f 100644 --- a/tests/qemuxml2argvmock.c +++ b/tests/qemuxml2argvmock.c @@ -205,7 +205,7 @@ virHostGetDRMRenderNode(void)
static void (*real_virCommandPassFD)(virCommandPtr cmd, int fd, unsigned int flags);
-static const int testCommandPassSafeFDs[] = { 1730, 1731 }; +static const int testCommandPassSafeFDs[] = { 1730, 1731, 1732 };
void virCommandPassFD(virCommandPtr cmd, @@ -283,3 +283,12 @@ qemuBuildTPMOpenBackendFDs(const char *tpmdev G_GNUC_UNUSED, *cancelfd = 1731; return 0; } + + +int +qemuInterfaceVDPAConnect(virDomainNetDefPtr net G_GNUC_UNUSED) +{ + if (fcntl(1732, F_GETFD) != -1) + abort(); + return 1732; +} diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index e93948e3fc..000d6919f2 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -1446,6 +1446,7 @@ mymain(void) QEMU_CAPS_DEVICE_VFIO_PCI); DO_TEST_FAILURE("net-hostdev-fail", QEMU_CAPS_DEVICE_VFIO_PCI); + DO_TEST_CAPS_LATEST("net-vdpa");
DO_TEST("hostdev-pci-multifunction", QEMU_CAPS_KVM, diff --git a/tests/qemuxml2xmloutdata/net-vdpa.xml b/tests/qemuxml2xmloutdata/net-vdpa.xml new file mode 100644 index 0000000000..b362405c14 --- /dev/null +++ b/tests/qemuxml2xmloutdata/net-vdpa.xml @@ -0,0 +1,34 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='i686' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-i386</emulator> + <controller type='usb' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='ide' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <interface type='vdpa'> + <mac address='52:54:00:95:db:c0'/> + <source dev='/dev/vhost-vdpa-0'/> + <model type='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </interface> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c index 39a9da874f..ac33e87698 100644 --- a/tests/qemuxml2xmltest.c +++ b/tests/qemuxml2xmltest.c @@ -497,6 +497,7 @@ mymain(void) DO_TEST("net-mtu", NONE); DO_TEST("net-coalesce", NONE); DO_TEST("net-many-models", NONE); + DO_TEST("net-vdpa", QEMU_CAPS_NETDEV_VHOST_VDPA);
DO_TEST("serial-tcp-tlsx509-chardev", NONE); DO_TEST("serial-tcp-tlsx509-chardev-notls", NONE);

On 9/11/20 5:17 PM, Jonathon Jongsma wrote:
vDPA network devices allow high-performance networking in a virtual machine by providing a wire-speed data path. These devices require a vendor-specific host driver but the data path follows the virtio specification.
The support for vDPA devices was recently added to qemu. This allows libvirt to support these devices. This patchset requires that the device is configured on the host with the appropriate vendor-specific driver. This will create a chardev on the host at e.g. /dev/vhost-vdpa-0. That chardev path can then be used to define a new interface with type='vdpa'.
Changes in v3: - rebased to latest master - various small fixes per comments on last review - investigated hotplug -- I have a preliminary libvirt patch, but it doesn't appear that hotplug is fully supported in qemu yet. - qemu never closes the fd when the device is removed, so the device cannot be reassigned to a different domain. - I decided not to post the preliminary hotplug patch until I can get more clarity from qemu developers.
This all looks good to me, and it passed full gitlab CI. I think it's okay to push without hotplug working (i.e. without the patch to enable it in libvirt) as long as the *intent* of qemu is that it is going to work. I don't remember what the end of the discussion about migration was - if vdpa itself doesn't support migration now, then obviously the current code is the right thing to do. They should keep in mind that when they *do* make the devices migratable, there will need to be a way for libvirt to discover that. I would be okay with pushing this, as long as it has been tested on real world equipment. Assuming that (but waiting for a response): Reviewed-by: Laine Stump <laine@redhat.com> (modulo a couple error log message wordings in patch 3 - if you end up sending the series again, it would help to have those in there. If we push this version, then make sure to remind me (or whoever pushes it) to make those tiny changes.) (BTW, didn't we have a discussion about adding reporting of vdpa devices to the node device driver?)
Jonathon Jongsma (3): conf: Add support for vDPA network devices qemu: add vhost-vdpa capability qemu: add vdpa support
docs/formatdomain.rst | 24 ++++++++++++ docs/schemas/domaincommon.rng | 15 ++++++++ src/conf/domain_conf.c | 31 ++++++++++++++++ src/conf/domain_conf.h | 4 ++ src/conf/netdev_bandwidth_conf.c | 1 + src/libxl/libxl_conf.c | 1 + src/libxl/xen_common.c | 1 + src/lxc/lxc_controller.c | 1 + src/lxc/lxc_driver.c | 3 ++ src/lxc/lxc_process.c | 1 + src/qemu/qemu_capabilities.c | 4 ++ src/qemu/qemu_capabilities.h | 3 ++ src/qemu/qemu_command.c | 31 +++++++++++++++- src/qemu/qemu_command.h | 3 +- src/qemu/qemu_domain.c | 6 ++- src/qemu/qemu_hotplug.c | 15 +++++--- src/qemu/qemu_interface.c | 25 +++++++++++++ src/qemu/qemu_interface.h | 2 + src/qemu/qemu_migration.c | 10 ++++- src/qemu/qemu_process.c | 2 + src/qemu/qemu_validate.c | 15 ++++++++ src/vmx/vmx.c | 1 + .../caps_5.1.0.x86_64.xml | 1 + .../caps_5.2.0.x86_64.xml | 1 + .../net-vdpa.x86_64-latest.args | 37 +++++++++++++++++++ tests/qemuxml2argvdata/net-vdpa.xml | 28 ++++++++++++++ tests/qemuxml2argvmock.c | 11 +++++- tests/qemuxml2argvtest.c | 1 + tests/qemuxml2xmloutdata/net-vdpa.xml | 34 +++++++++++++++++ tests/qemuxml2xmltest.c | 1 + tools/virsh-domain.c | 1 + 31 files changed, 303 insertions(+), 11 deletions(-) create mode 100644 tests/qemuxml2argvdata/net-vdpa.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/net-vdpa.xml create mode 100644 tests/qemuxml2xmloutdata/net-vdpa.xml
participants (2)
-
Jonathon Jongsma
-
Laine Stump