[libvirt] [PATCH v2 0/7] PCI Multifunction hotplug/unplug, part 1
v2: - rebased without 3 patches from Michal that are under review in his NVMe patch series This is the first part of the feature discussed at [1]. These patches are mostly cleanup and fixes, thus it is beneficial to have them upstream right away. The whole feature can be checked out at [2]. All patches survives unit testing. The feature was stress tested with hundreds of consecutive hotplug/unplugs of a Broadcom BCM5719 multifunction network card in a guest running in a Power 8 server. Hopefully I'll find a suitable x86 env to stress test the feature there too. [1] https://www.redhat.com/archives/libvir-list/2019-June/msg00703.html [2] https://github.com/danielhb/libvirt/tree/multifunction_latest Daniel Henrique Barboza (1): util/virhostdev: enhance VFIO device is in use detection Shivaprasad G Bhat (6): tests: Fix the iommu group path in mock pci tests: pci: Mock the iommu groups and vfio virpcitest: Change the stub driver to vfio from pci-stub virpcimock: Mock the SRIOV Virtual functions tests: qemu: Add test case for pci-hostdev hotplug tests: Add a baseline test for multifunction pci device use case src/util/virhostdev.c | 74 +++++-- src/util/virprocess.h | 2 +- tests/Makefile.am | 7 + tests/qemuhotplugtest.c | 42 +++- .../qemuhotplug-hostdev-pci.xml | 6 + .../qemuhotplug-base-live+hostdev-pci.xml | 58 +++++ ...uhotplug-pseries-base-live+hostdev-pci.xml | 51 +++++ .../qemuhotplug-pseries-base-live.xml | 43 ++++ .../hostdev-pci-multifunction.args | 35 +++ .../hostdev-pci-multifunction.xml | 59 +++++ .../hostdev-vfio-multidomain.args | 2 +- .../hostdev-vfio-multidomain.xml | 2 +- tests/qemuxml2argvdata/hostdev-vfio.args | 2 +- tests/qemuxml2argvdata/hostdev-vfio.xml | 2 +- tests/qemuxml2argvdata/net-hostdev-fail.xml | 2 +- tests/qemuxml2argvdata/net-hostdev-vfio.args | 2 +- tests/qemuxml2argvdata/net-hostdev-vfio.xml | 2 +- tests/qemuxml2argvtest.c | 3 + .../hostdev-pci-multifunction.xml | 79 +++++++ tests/qemuxml2xmloutdata/hostdev-vfio.xml | 2 +- tests/qemuxml2xmloutdata/net-hostdev-vfio.xml | 2 +- tests/qemuxml2xmltest.c | 1 + tests/virhostdevtest.c | 4 +- tests/virpcimock.c | 201 ++++++++++++++++-- tests/virpcitest.c | 12 +- tests/virpcitestdata/0000-06-12.0.config | Bin 0 -> 256 bytes tests/virpcitestdata/0000-06-12.1.config | Bin 0 -> 256 bytes tests/virpcitestdata/0000-06-12.2.config | Bin 0 -> 256 bytes tests/virprocessmock.c | 28 +++ 29 files changed, 668 insertions(+), 55 deletions(-) create mode 100644 tests/qemuhotplugtestdevices/qemuhotplug-hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-base-live+hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live+hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live.xml create mode 100644 tests/qemuxml2argvdata/hostdev-pci-multifunction.args create mode 100644 tests/qemuxml2argvdata/hostdev-pci-multifunction.xml create mode 100644 tests/qemuxml2xmloutdata/hostdev-pci-multifunction.xml create mode 100644 tests/virpcitestdata/0000-06-12.0.config create mode 100644 tests/virpcitestdata/0000-06-12.1.config create mode 100644 tests/virpcitestdata/0000-06-12.2.config create mode 100644 tests/virprocessmock.c -- 2.21.0
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> The mocked path falls into /sys/bus/kernel/iommu_groups instead of /sys/kernel/iommu_groups. Needed for adding some new test cases. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/virpcimock.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/virpcimock.c b/tests/virpcimock.c index beb5e1490d..5ec9abe05f 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -399,7 +399,7 @@ pci_device_new_from_stub(const struct pciDevice *data) make_file(devpath, "class", tmp, -1); if (snprintf(tmp, sizeof(tmp), - "%s/../../../kernel/iommu_groups/%d", + "%s/../../../../kernel/iommu_groups/%d", devpath, dev->iommuGroup) < 0) { ABORT("@tmp overflow"); } @@ -407,7 +407,7 @@ pci_device_new_from_stub(const struct pciDevice *data) ABORT("Unable to create %s", tmp); if (snprintf(tmp, sizeof(tmp), - "../../../kernel/iommu_groups/%d", dev->iommuGroup) < 0) { + "../../../../kernel/iommu_groups/%d", dev->iommuGroup) < 0) { ABORT("@tmp overflow"); } make_symlink(devpath, "iommu_group", tmp); -- 2.21.0
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com>
The mocked path falls into /sys/bus/kernel/iommu_groups instead of /sys/kernel/iommu_groups. Needed for adding some new test cases.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/virpcimock.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/virpcimock.c b/tests/virpcimock.c index beb5e1490d..5ec9abe05f 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -399,7 +399,7 @@ pci_device_new_from_stub(const struct pciDevice *data) make_file(devpath, "class", tmp, -1);
if (snprintf(tmp, sizeof(tmp), - "%s/../../../kernel/iommu_groups/%d", + "%s/../../../../kernel/iommu_groups/%d", devpath, dev->iommuGroup) < 0) { ABORT("@tmp overflow"); } @@ -407,7 +407,7 @@ pci_device_new_from_stub(const struct pciDevice *data) ABORT("Unable to create %s", tmp);
if (snprintf(tmp, sizeof(tmp), - "../../../kernel/iommu_groups/%d", dev->iommuGroup) < 0) { + "../../../../kernel/iommu_groups/%d", dev->iommuGroup) < 0) { ABORT("@tmp overflow"); } make_symlink(devpath, "iommu_group", tmp);
Huh, this is again something that I fix in my series. Well, in fact, since I'm moving the place where PCI devices are created the symlink as we have it is correct in fact. Michal
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> The iommu group, /dev/vfio/<iommuGroupNumber> behaviors of the host are mocked. This patch implements support for multifunction/multiple devices per iommu groups and emulates the /dev/vfio/<iommuGroupNumber> file correctly. This code helps adding necessary testcases for pci-hotplug code. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/virpcimock.c | 177 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 162 insertions(+), 15 deletions(-) diff --git a/tests/virpcimock.c b/tests/virpcimock.c index 5ec9abe05f..1b44ed0a44 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -44,6 +44,8 @@ char *fakerootdir; char *fakesysfspcidir; # define SYSFS_PCI_PREFIX "/sys/bus/pci/" +# define SYSFS_KERNEL_PREFIX "/sys/kernel/" + # define STDERR(...) \ fprintf(stderr, "%s %zu: ", __FUNCTION__, (size_t) __LINE__); \ @@ -123,6 +125,11 @@ struct pciDevice { struct pciDriver *driver; /* Driver attached. NULL if attached to no driver */ }; +struct pciIommuGroup { + int iommu; + size_t nDevicesBoundToVFIO; /* Indicates the devices in the group */ +}; + struct fdCallback { int fd; char *path; @@ -134,6 +141,9 @@ size_t nPCIDevices = 0; struct pciDriver **pciDrivers = NULL; size_t nPCIDrivers = 0; +struct pciIommuGroup **pciIommuGroups = NULL; +size_t npciIommuGroups = 0; + struct fdCallback *callbacks = NULL; size_t nCallbacks = 0; @@ -247,6 +257,13 @@ getrealpath(char **newpath, errno = ENOMEM; return -1; } + } else if (STRPREFIX(path, SYSFS_KERNEL_PREFIX)) { + if (virAsprintfQuiet(newpath, "%s/%s", + fakerootdir, + path) < 0) { + errno = ENOMEM; + return -1; + } } else { if (VIR_STRDUP_QUIET(*newpath, path) < 0) return -1; @@ -460,6 +477,101 @@ pci_device_autobind(struct pciDevice *dev) return pci_driver_bind(driver, dev); } +static void +pci_iommu_new(int num) +{ + char *iommupath, *kerneldir; + struct pciIommuGroup *iommuGroup; + + if (VIR_ALLOC_QUIET(iommuGroup) < 0) + ABORT_OOM(); + + iommuGroup->iommu = num; + iommuGroup->nDevicesBoundToVFIO = 0; /* No device bound to vfio by default */ + + if (virAsprintfQuiet(&kerneldir, "%s%s", + fakerootdir, SYSFS_KERNEL_PREFIX) < 0) + ABORT_OOM(); + + if (virAsprintfQuiet(&iommupath, "%s/iommu_groups/%d/devices", kerneldir, num) < 0) + ABORT_OOM(); + VIR_FREE(kerneldir); + + if (virFileMakePath(iommupath) < 0) + ABORT("Unable to create: %s", iommupath); + VIR_FREE(iommupath); + + if (VIR_APPEND_ELEMENT_QUIET(pciIommuGroups, npciIommuGroups, iommuGroup) < 0) + ABORT_OOM(); +} + +static int +pci_vfio_release_iommu(struct pciDevice *device) +{ + char *vfiopath = NULL; + int ret = -1; + size_t i = 0; + + for (i = 0; i < npciIommuGroups; i++) { + if (pciIommuGroups[i]->iommu == device->iommuGroup) + break; + } + + if (i != npciIommuGroups) { + if (pciIommuGroups[i]->nDevicesBoundToVFIO == 0) { + ret = 0; + goto cleanup; + } + pciIommuGroups[i]->nDevicesBoundToVFIO--; + if (!pciIommuGroups[i]->nDevicesBoundToVFIO) { + if (virAsprintfQuiet(&vfiopath, "%s/dev/vfio/%d", + fakesysfspcidir, device->iommuGroup) < 0) { + errno = ENOMEM; + goto cleanup; + } + if (unlink(vfiopath) < 0) + goto cleanup; + } + } + + ret = 0; + cleanup: + VIR_FREE(vfiopath); + return ret; +} + +static int +pci_vfio_lock_iommu(struct pciDevice *device) +{ + char *vfiopath = NULL; + int ret = -1; + size_t i = 0; + int fd = -1; + + for (i = 0; i < npciIommuGroups; i++) { + if (pciIommuGroups[i]->iommu == device->iommuGroup) + break; + } + + if (i != npciIommuGroups) { + if (!pciIommuGroups[i]->nDevicesBoundToVFIO) { + if (virAsprintfQuiet(&vfiopath, "%s/dev/vfio/%d", + fakesysfspcidir, device->iommuGroup) < 0) { + errno = ENOMEM; + goto cleanup; + } + if ((fd = real_open(vfiopath, O_CREAT)) < 0) + goto cleanup; + } + pciIommuGroups[i]->nDevicesBoundToVFIO++; + } + + ret = 0; + cleanup: + real_close(fd); + VIR_FREE(vfiopath); + return ret; +} /* * PCI Driver functions @@ -583,6 +695,10 @@ pci_driver_bind(struct pciDriver *driver, if (symlink(devpath, driverpath) < 0) goto cleanup; + if (STREQ(driver->name, "vfio-pci")) + if (pci_vfio_lock_iommu(dev) < 0) + goto cleanup; + dev->driver = driver; ret = 0; cleanup: @@ -617,6 +733,10 @@ pci_driver_unbind(struct pciDriver *driver, unlink(driverpath) < 0) goto cleanup; + if (STREQ(driver->name, "vfio-pci")) + if (pci_vfio_release_iommu(dev) < 0) + goto cleanup; + dev->driver = NULL; ret = 0; cleanup: @@ -813,6 +933,8 @@ init_syms(void) static void init_env(void) { + char *devVFIO; + if (fakerootdir && fakesysfspcidir) return; @@ -828,6 +950,29 @@ init_env(void) make_file(fakesysfspcidir, "drivers_probe", NULL, -1); + if (virAsprintfQuiet(&devVFIO, "%s/dev/vfio", fakesysfspcidir) < 0) + ABORT_OOM(); + + if (virFileMakePath(devVFIO) < 0) + ABORT("Unable to create: %s", devVFIO); + + /* Create /dev/vfio/vfio file */ + make_file(devVFIO, "vfio", NULL, -1); + VIR_FREE(devVFIO); + + pci_iommu_new(0); + pci_iommu_new(1); + pci_iommu_new(2); + pci_iommu_new(3); + pci_iommu_new(4); + pci_iommu_new(5); + pci_iommu_new(6); + pci_iommu_new(7); + pci_iommu_new(8); + pci_iommu_new(9); + pci_iommu_new(10); + pci_iommu_new(11); + # define MAKE_PCI_DRIVER(name, ...) \ pci_driver_new(name, 0, __VA_ARGS__, -1, -1) @@ -843,20 +988,21 @@ init_env(void) pci_device_new_from_stub(&dev); \ } while (0) - MAKE_PCI_DEVICE("0000:00:00.0", 0x8086, 0x0044); - MAKE_PCI_DEVICE("0000:00:01.0", 0x8086, 0x0044); - MAKE_PCI_DEVICE("0000:00:02.0", 0x8086, 0x0046); - MAKE_PCI_DEVICE("0000:00:03.0", 0x8086, 0x0048); - MAKE_PCI_DEVICE("0001:00:00.0", 0x1014, 0x03b9, .klass = 0x060400); - MAKE_PCI_DEVICE("0001:01:00.0", 0x8086, 0x105e, .iommuGroup = 0); - MAKE_PCI_DEVICE("0001:01:00.1", 0x8086, 0x105e, .iommuGroup = 0); - MAKE_PCI_DEVICE("0005:80:00.0", 0x10b5, 0x8112, .klass = 0x060400); - MAKE_PCI_DEVICE("0005:90:01.0", 0x1033, 0x0035, .iommuGroup = 1); - MAKE_PCI_DEVICE("0005:90:01.1", 0x1033, 0x0035, .iommuGroup = 1); - MAKE_PCI_DEVICE("0005:90:01.2", 0x1033, 0x00e0, .iommuGroup = 1); - MAKE_PCI_DEVICE("0000:0a:01.0", 0x8086, 0x0047); - MAKE_PCI_DEVICE("0000:0a:02.0", 0x8286, 0x0048); - MAKE_PCI_DEVICE("0000:0a:03.0", 0x8386, 0x0048); + MAKE_PCI_DEVICE("0000:00:00.0", 0x8086, 0x0044, .iommuGroup = 0); + MAKE_PCI_DEVICE("0000:00:01.0", 0x8086, 0x0044, .iommuGroup = 1); + MAKE_PCI_DEVICE("0000:00:02.0", 0x8086, 0x0046, .iommuGroup = 2); + MAKE_PCI_DEVICE("0000:00:03.0", 0x8086, 0x0048, .iommuGroup = 3); + MAKE_PCI_DEVICE("0001:00:00.0", 0x1014, 0x03b9, .klass = 0x060400, .iommuGroup = 4); + MAKE_PCI_DEVICE("0001:01:00.0", 0x8086, 0x105e, .iommuGroup = 5); + MAKE_PCI_DEVICE("0001:01:00.1", 0x8086, 0x105e, .iommuGroup = 5); + MAKE_PCI_DEVICE("0005:80:00.0", 0x10b5, 0x8112, .klass = 0x060400, .iommuGroup = 6); + MAKE_PCI_DEVICE("0005:90:01.0", 0x1033, 0x0035, .iommuGroup = 7); + MAKE_PCI_DEVICE("0005:90:01.1", 0x1033, 0x0035, .iommuGroup = 7); + MAKE_PCI_DEVICE("0005:90:01.2", 0x1033, 0x00e0, .iommuGroup = 7); + MAKE_PCI_DEVICE("0005:90:01.3", 0x1033, 0x00e0, .iommuGroup = 7); + MAKE_PCI_DEVICE("0000:0a:01.0", 0x8086, 0x0047, .iommuGroup = 8); + MAKE_PCI_DEVICE("0000:0a:02.0", 0x8286, 0x0048, .iommuGroup = 8); + MAKE_PCI_DEVICE("0000:0a:03.0", 0x8386, 0x0048, .iommuGroup = 8); } @@ -940,7 +1086,8 @@ opendir(const char *path) init_syms(); - if (STRPREFIX(path, SYSFS_PCI_PREFIX) && + if ((STRPREFIX(path, SYSFS_PCI_PREFIX) || + STRPREFIX(path, SYSFS_KERNEL_PREFIX)) && getrealpath(&newpath, path) < 0) return NULL; -- 2.21.0
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com>
The iommu group, /dev/vfio/<iommuGroupNumber> behaviors of the host are mocked. This patch implements support for multifunction/multiple devices per iommu groups and emulates the /dev/vfio/<iommuGroupNumber> file correctly.
This code helps adding necessary testcases for pci-hotplug code.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/virpcimock.c | 177 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 162 insertions(+), 15 deletions(-)
Looks good. We will have some merge conflicts because we fix the same issue.
diff --git a/tests/virpcimock.c b/tests/virpcimock.c index 5ec9abe05f..1b44ed0a44 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -44,6 +44,8 @@ char *fakerootdir; char *fakesysfspcidir;
# define SYSFS_PCI_PREFIX "/sys/bus/pci/" +# define SYSFS_KERNEL_PREFIX "/sys/kernel/" +
# define STDERR(...) \ fprintf(stderr, "%s %zu: ", __FUNCTION__, (size_t) __LINE__); \ @@ -123,6 +125,11 @@ struct pciDevice { struct pciDriver *driver; /* Driver attached. NULL if attached to no driver */ };
+struct pciIommuGroup { + int iommu; + size_t nDevicesBoundToVFIO; /* Indicates the devices in the group */ +}; + struct fdCallback { int fd; char *path; @@ -134,6 +141,9 @@ size_t nPCIDevices = 0; struct pciDriver **pciDrivers = NULL; size_t nPCIDrivers = 0;
+struct pciIommuGroup **pciIommuGroups = NULL; +size_t npciIommuGroups = 0; + struct fdCallback *callbacks = NULL; size_t nCallbacks = 0;
@@ -247,6 +257,13 @@ getrealpath(char **newpath, errno = ENOMEM; return -1; } + } else if (STRPREFIX(path, SYSFS_KERNEL_PREFIX)) { + if (virAsprintfQuiet(newpath, "%s/%s", + fakerootdir, + path) < 0) { + errno = ENOMEM; + return -1; + } } else { if (VIR_STRDUP_QUIET(*newpath, path) < 0) return -1; @@ -460,6 +477,101 @@ pci_device_autobind(struct pciDevice *dev) return pci_driver_bind(driver, dev); }
+static void +pci_iommu_new(int num) +{ + char *iommupath, *kerneldir; + struct pciIommuGroup *iommuGroup; + + if (VIR_ALLOC_QUIET(iommuGroup) < 0) + ABORT_OOM(); + + iommuGroup->iommu = num; + iommuGroup->nDevicesBoundToVFIO = 0; /* No device bound to vfio by default */ + + if (virAsprintfQuiet(&kerneldir, "%s%s", + fakerootdir, SYSFS_KERNEL_PREFIX) < 0) + ABORT_OOM(); + + if (virAsprintfQuiet(&iommupath, "%s/iommu_groups/%d/devices", kerneldir, num) < 0) + ABORT_OOM(); + VIR_FREE(kerneldir); + + if (virFileMakePath(iommupath) < 0) + ABORT("Unable to create: %s", iommupath); + VIR_FREE(iommupath); + + if (VIR_APPEND_ELEMENT_QUIET(pciIommuGroups, npciIommuGroups, iommuGroup) < 0) + ABORT_OOM(); +} + +static int +pci_vfio_release_iommu(struct pciDevice *device) +{ + char *vfiopath = NULL; + int ret = -1; + size_t i = 0; + + for (i = 0; i < npciIommuGroups; i++) { + if (pciIommuGroups[i]->iommu == device->iommuGroup) + break; + } + + if (i != npciIommuGroups) { + if (pciIommuGroups[i]->nDevicesBoundToVFIO == 0) { + ret = 0; + goto cleanup; + } + pciIommuGroups[i]->nDevicesBoundToVFIO--; + if (!pciIommuGroups[i]->nDevicesBoundToVFIO) { + if (virAsprintfQuiet(&vfiopath, "%s/dev/vfio/%d", + fakesysfspcidir, device->iommuGroup) < 0) { + errno = ENOMEM; + goto cleanup; + } + if (unlink(vfiopath) < 0) + goto cleanup; + } + } + + ret = 0; + cleanup: + VIR_FREE(vfiopath); + return ret; +} + +static int +pci_vfio_lock_iommu(struct pciDevice *device) +{ + char *vfiopath = NULL; + int ret = -1; + size_t i = 0; + int fd = -1; + + for (i = 0; i < npciIommuGroups; i++) { + if (pciIommuGroups[i]->iommu == device->iommuGroup) + break; + } + + if (i != npciIommuGroups) { + if (!pciIommuGroups[i]->nDevicesBoundToVFIO) { + if (virAsprintfQuiet(&vfiopath, "%s/dev/vfio/%d", + fakesysfspcidir, device->iommuGroup) < 0) { + errno = ENOMEM; + goto cleanup; + } + if ((fd = real_open(vfiopath, O_CREAT)) < 0) + goto cleanup; + } + pciIommuGroups[i]->nDevicesBoundToVFIO++; + } + + ret = 0; + cleanup: + real_close(fd); + VIR_FREE(vfiopath); + return ret; +}
/* * PCI Driver functions @@ -583,6 +695,10 @@ pci_driver_bind(struct pciDriver *driver, if (symlink(devpath, driverpath) < 0) goto cleanup;
+ if (STREQ(driver->name, "vfio-pci")) + if (pci_vfio_lock_iommu(dev) < 0) + goto cleanup; + dev->driver = driver; ret = 0; cleanup: @@ -617,6 +733,10 @@ pci_driver_unbind(struct pciDriver *driver, unlink(driverpath) < 0) goto cleanup;
+ if (STREQ(driver->name, "vfio-pci")) + if (pci_vfio_release_iommu(dev) < 0) + goto cleanup; + dev->driver = NULL; ret = 0; cleanup: @@ -813,6 +933,8 @@ init_syms(void) static void init_env(void) { + char *devVFIO; + if (fakerootdir && fakesysfspcidir) return;
@@ -828,6 +950,29 @@ init_env(void)
make_file(fakesysfspcidir, "drivers_probe", NULL, -1);
+ if (virAsprintfQuiet(&devVFIO, "%s/dev/vfio", fakesysfspcidir) < 0) + ABORT_OOM(); + + if (virFileMakePath(devVFIO) < 0) + ABORT("Unable to create: %s", devVFIO); + + /* Create /dev/vfio/vfio file */ + make_file(devVFIO, "vfio", NULL, -1); + VIR_FREE(devVFIO); + + pci_iommu_new(0); + pci_iommu_new(1); + pci_iommu_new(2); + pci_iommu_new(3); + pci_iommu_new(4); + pci_iommu_new(5); + pci_iommu_new(6); + pci_iommu_new(7); + pci_iommu_new(8); + pci_iommu_new(9); + pci_iommu_new(10); + pci_iommu_new(11);
I wonder if we can make these somehow dyamic. E.g. called from pci_device_new_from_stub() and require iommuGroup in MAKE_PCI_DEVICE() because every device belongs to an IOMMU group. Michal
The current virHostdevPreparePCIDevices code fails to detect an unmanaged VFIO device that is in the activePCIHostdevs, and active in the same domain name as dom_name, as a device in use. Considering a call to this function, when activePCIHostdevs has a VFIO deviceA in the list, and deviceA is active in domain 'dom_name', this is what happens: - At step 1, the code goes to the 'if (usesVFIO)' block. All devices in the same IOMMU group of deviceA are used as argument of virHostdevIsPCINodeDeviceUsed, via the IOMMUGroupIterate function; - Inside virHostdevIsPCINodeDeviceUsed, an 'usesVFIO' verification is made, following to an 'iommu_owner' jump that sets ret=0. This will happen to all devices of the IOMMU group that belongs to the same domain as dom_name, including deviceA; - Step 2 starts, thinking that all devices inside hostdevs are available, which is not the case. This error was detected when changing tests/virhostdev.c to use vfio-pci instead of pci-stub (a change that will follow this one). The side effect observed was a test failure in testVirHostdevPreparePCIHostdevs_unmanaged, result of the behavior mentioned above: Unexpected count of items in mgr->inactivePCIHostdevs: 1, expecting 0 This patch fixes virHostdevIsPCINodeDeviceUsed to consider the case of a VFIO device that is already active in the domain. First, check if the device is in the activePCIHostdev list and bail immediately if true. Otherwise, in case of VFIO, check for IOMMU group ownership of the domain. This is done by a new callback function for IOMMUGroupIterate. If the VFIO device isn't in the active list and its domain has ownership of the IOMMU, then it is unused. Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- src/util/virhostdev.c | 74 +++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 17 deletions(-) diff --git a/src/util/virhostdev.c b/src/util/virhostdev.c index a3647a6cf4..35be8fede1 100644 --- a/src/util/virhostdev.c +++ b/src/util/virhostdev.c @@ -55,6 +55,46 @@ struct virHostdevIsPCINodeDeviceUsedData { const bool usesVFIO; }; + +static virPCIDevicePtr +virHostdevFindActivePCIDevWithAddr(virPCIDeviceAddressPtr devAddr, + virHostdevManagerPtr mgr) +{ + virPCIDevicePtr ret = NULL; + + ret = virPCIDeviceListFindByIDs(mgr->activePCIHostdevs, + devAddr->domain, devAddr->bus, + devAddr->slot, devAddr->function); + + return ret; +} + +/* Callback to be used inside virHostdevIsPCINodeDeviceUsed to check + * for IOMMU ownership of a domain given by helperData->domainName. */ +static int +virHostdevDomainHasIOMMUOwnershipCb(virPCIDeviceAddressPtr devAddr, void *opaque) +{ + struct virHostdevIsPCINodeDeviceUsedData *helperData = opaque; + virPCIDevicePtr actual; + int ret = 0; + + actual = virHostdevFindActivePCIDevWithAddr(devAddr, helperData->mgr); + if (!actual) + return ret; + + if (helperData->usesVFIO) { + const char *actual_drvname = NULL; + const char *actual_domname = NULL; + virPCIDeviceGetUsedBy(actual, &actual_drvname, &actual_domname); + + if ((actual_domname && helperData->domainName) && + (STRNEQ(actual_domname, helperData->domainName))) + ret = -1; + } + + return ret; +} + /* This module makes heavy use of bookkeeping lists contained inside a * virHostdevManager instance to keep track of the devices' status. To make * it easy to spot potential ownership errors when moving devices from one @@ -82,19 +122,13 @@ static int virHostdevIsPCINodeDeviceUsed(virPCIDeviceAddressPtr devAddr, void *o int ret = -1; struct virHostdevIsPCINodeDeviceUsedData *helperData = opaque; - actual = virPCIDeviceListFindByIDs(helperData->mgr->activePCIHostdevs, - devAddr->domain, devAddr->bus, - devAddr->slot, devAddr->function); + actual = virHostdevFindActivePCIDevWithAddr(devAddr, helperData->mgr); + if (actual) { const char *actual_drvname = NULL; const char *actual_domname = NULL; virPCIDeviceGetUsedBy(actual, &actual_drvname, &actual_domname); - if (helperData->usesVFIO && - (actual_domname && helperData->domainName) && - (STREQ(actual_domname, helperData->domainName))) - goto iommu_owner; - if (actual_drvname && actual_domname) virReportError(VIR_ERR_OPERATION_INVALID, _("PCI device %s is in use by " @@ -105,9 +139,20 @@ static int virHostdevIsPCINodeDeviceUsed(virPCIDeviceAddressPtr devAddr, void *o virReportError(VIR_ERR_OPERATION_INVALID, _("PCI device %s is in use"), virPCIDeviceGetName(actual)); + goto cleanup; } - iommu_owner: + + /* For VFIO devices, the domain helperData->domainName must have ownership + * of the IOMMU group that contains devAddr. Otherwise, even if the devAddr + * is not in use, another device of that IOMMU group is in use by anotherdomain, + * forbidding devAddr to be used in helperData->domainName. */ + if (helperData->usesVFIO) + if (virPCIDeviceAddressIOMMUGroupIterate(devAddr, + virHostdevDomainHasIOMMUOwnershipCb, + helperData) < 0) + goto cleanup; + ret = 0; cleanup: return ret; @@ -673,17 +718,12 @@ virHostdevPreparePCIDevices(virHostdevManagerPtr mgr, /* The device is in use by other active domain if * the dev is in list activePCIHostdevs. VFIO devices * belonging to same iommu group can't be shared - * across guests. + * across guests. virHostdevIsPCINodeDeviceUsedData handles + * both cases. */ devAddr = virPCIDeviceGetAddress(pci); - if (usesVFIO) { - if (virPCIDeviceAddressIOMMUGroupIterate(devAddr, - virHostdevIsPCINodeDeviceUsed, - &data) < 0) - goto cleanup; - } else if (virHostdevIsPCINodeDeviceUsed(devAddr, &data)) { + if (virHostdevIsPCINodeDeviceUsed(devAddr, &data)) goto cleanup; - } } /* Step 1.5: For non-802.11Qbh SRIOV network devices, save the -- 2.21.0
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
The current virHostdevPreparePCIDevices code fails to detect an unmanaged VFIO device that is in the activePCIHostdevs, and active in the same domain name as dom_name, as a device in use. Considering a call to this function, when activePCIHostdevs has a VFIO deviceA in the list, and deviceA is active in domain 'dom_name', this is what happens:
- At step 1, the code goes to the 'if (usesVFIO)' block. All devices in the same IOMMU group of deviceA are used as argument of virHostdevIsPCINodeDeviceUsed, via the IOMMUGroupIterate function;
- Inside virHostdevIsPCINodeDeviceUsed, an 'usesVFIO' verification is made, following to an 'iommu_owner' jump that sets ret=0. This will happen to all devices of the IOMMU group that belongs to the same domain as dom_name, including deviceA;
- Step 2 starts, thinking that all devices inside hostdevs are available, which is not the case.
This error was detected when changing tests/virhostdev.c to use vfio-pci instead of pci-stub (a change that will follow this one). The side effect observed was a test failure in testVirHostdevPreparePCIHostdevs_unmanaged, result of the behavior mentioned above:
Unexpected count of items in mgr->inactivePCIHostdevs: 1, expecting 0
This patch fixes virHostdevIsPCINodeDeviceUsed to consider the case of a VFIO device that is already active in the domain. First, check if the device is in the activePCIHostdev list and bail immediately if true. Otherwise, in case of VFIO, check for IOMMU group ownership of the domain. This is done by a new callback function for IOMMUGroupIterate. If the VFIO device isn't in the active list and its domain has ownership of the IOMMU, then it is unused.
Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- src/util/virhostdev.c | 74 +++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 17 deletions(-)
As you pointed out, I've sent a patch for the same issue. And if you don't mind, I'd like to go with mine, because it's shorter and simpler. Michal
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> The pci-stub is obsolete for a while now. Upcoming test cases try to test the VFIO hotplug/unplug cases. Change the default test driver to vfio-pci instead of pci-stub, and fail bind for pci-stub instead. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/virhostdevtest.c | 4 ++-- tests/virpcimock.c | 4 ++-- tests/virpcitest.c | 12 ++++++------ 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/tests/virhostdevtest.c b/tests/virhostdevtest.c index 20eaca82e0..99ee2d44ec 100644 --- a/tests/virhostdevtest.c +++ b/tests/virhostdevtest.c @@ -93,7 +93,7 @@ myInit(void) subsys.u.pci.addr.bus = 0; subsys.u.pci.addr.slot = i + 1; subsys.u.pci.addr.function = 0; - subsys.u.pci.backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_KVM; + subsys.u.pci.backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VFIO; hostdevs[i]->source.subsys = subsys; } @@ -101,7 +101,7 @@ myInit(void) if (!(dev[i] = virPCIDeviceNew(0, 0, i + 1, 0))) goto cleanup; - virPCIDeviceSetStubDriver(dev[i], VIR_PCI_STUB_DRIVER_KVM); + virPCIDeviceSetStubDriver(dev[i], VIR_PCI_STUB_DRIVER_VFIO); } if (VIR_ALLOC(mgr) < 0) diff --git a/tests/virpcimock.c b/tests/virpcimock.c index 1b44ed0a44..deb802e860 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -978,8 +978,8 @@ init_env(void) MAKE_PCI_DRIVER("iwlwifi", 0x8086, 0x0044); MAKE_PCI_DRIVER("i915", 0x8086, 0x0046, 0x8086, 0x0047); - MAKE_PCI_DRIVER("pci-stub", -1, -1); - pci_driver_new("vfio-pci", PCI_ACTION_BIND, -1, -1); + pci_driver_new("pci-stub", PCI_ACTION_BIND, -1, -1); + MAKE_PCI_DRIVER("vfio-pci", -1, -1); # define MAKE_PCI_DEVICE(Id, Vendor, Device, ...) \ do { \ diff --git a/tests/virpcitest.c b/tests/virpcitest.c index 961a7eff1a..d120e7773a 100644 --- a/tests/virpcitest.c +++ b/tests/virpcitest.c @@ -106,12 +106,12 @@ testVirPCIDeviceDetach(const void *opaque ATTRIBUTE_UNUSED) if (!(dev[i] = virPCIDeviceNew(0, 0, i + 1, 0))) goto cleanup; - virPCIDeviceSetStubDriver(dev[i], VIR_PCI_STUB_DRIVER_KVM); + virPCIDeviceSetStubDriver(dev[i], VIR_PCI_STUB_DRIVER_VFIO); if (virPCIDeviceDetach(dev[i], activeDevs, inactiveDevs) < 0) goto cleanup; - if (testVirPCIDeviceCheckDriver(dev[i], "pci-stub") < 0) + if (testVirPCIDeviceCheckDriver(dev[i], "vfio-pci") < 0) goto cleanup; CHECK_LIST_COUNT(activeDevs, 0); @@ -245,7 +245,7 @@ testVirPCIDeviceDetachSingle(const void *opaque) if (!dev) goto cleanup; - virPCIDeviceSetStubDriver(dev, VIR_PCI_STUB_DRIVER_KVM); + virPCIDeviceSetStubDriver(dev, VIR_PCI_STUB_DRIVER_VFIO); if (virPCIDeviceDetach(dev, NULL, NULL) < 0) goto cleanup; @@ -267,7 +267,7 @@ testVirPCIDeviceDetachFail(const void *opaque) if (!dev) goto cleanup; - virPCIDeviceSetStubDriver(dev, VIR_PCI_STUB_DRIVER_VFIO); + virPCIDeviceSetStubDriver(dev, VIR_PCI_STUB_DRIVER_KVM); if (virPCIDeviceDetach(dev, NULL, NULL) < 0) { if (virTestGetVerbose() || virTestGetDebug()) @@ -278,7 +278,7 @@ testVirPCIDeviceDetachFail(const void *opaque) virReportError(VIR_ERR_INTERNAL_ERROR, "Attaching device %s to %s should have failed", virPCIDeviceGetName(dev), - virPCIStubDriverTypeToString(VIR_PCI_STUB_DRIVER_VFIO)); + virPCIStubDriverTypeToString(VIR_PCI_STUB_DRIVER_KVM)); } cleanup: @@ -437,7 +437,7 @@ mymain(void) /* Detach an unbound device */ DO_TEST_PCI_DRIVER(0, 0x0a, 2, 0, NULL); DO_TEST_PCI(testVirPCIDeviceDetachSingle, 0, 0x0a, 2, 0); - DO_TEST_PCI_DRIVER(0, 0x0a, 2, 0, "pci-stub"); + DO_TEST_PCI_DRIVER(0, 0x0a, 2, 0, "vfio-pci"); /* Reattach an unknown unbound device */ DO_TEST_PCI_DRIVER(0, 0x0a, 3, 0, NULL); -- 2.21.0
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com>
The pci-stub is obsolete for a while now. Upcoming test cases try to test the VFIO hotplug/unplug cases.
Change the default test driver to vfio-pci instead of pci-stub, and fail bind for pci-stub instead.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- tests/virhostdevtest.c | 4 ++-- tests/virpcimock.c | 4 ++-- tests/virpcitest.c | 12 ++++++------ 3 files changed, 10 insertions(+), 10 deletions(-)
Huh, again, same patches ...
diff --git a/tests/virhostdevtest.c b/tests/virhostdevtest.c index 20eaca82e0..99ee2d44ec 100644 --- a/tests/virhostdevtest.c +++ b/tests/virhostdevtest.c @@ -93,7 +93,7 @@ myInit(void) subsys.u.pci.addr.bus = 0; subsys.u.pci.addr.slot = i + 1; subsys.u.pci.addr.function = 0; - subsys.u.pci.backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_KVM; + subsys.u.pci.backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VFIO; hostdevs[i]->source.subsys = subsys; }
@@ -101,7 +101,7 @@ myInit(void) if (!(dev[i] = virPCIDeviceNew(0, 0, i + 1, 0))) goto cleanup;
- virPCIDeviceSetStubDriver(dev[i], VIR_PCI_STUB_DRIVER_KVM); + virPCIDeviceSetStubDriver(dev[i], VIR_PCI_STUB_DRIVER_VFIO); }
if (VIR_ALLOC(mgr) < 0) diff --git a/tests/virpcimock.c b/tests/virpcimock.c index 1b44ed0a44..deb802e860 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -978,8 +978,8 @@ init_env(void)
MAKE_PCI_DRIVER("iwlwifi", 0x8086, 0x0044); MAKE_PCI_DRIVER("i915", 0x8086, 0x0046, 0x8086, 0x0047); - MAKE_PCI_DRIVER("pci-stub", -1, -1); - pci_driver_new("vfio-pci", PCI_ACTION_BIND, -1, -1); + pci_driver_new("pci-stub", PCI_ACTION_BIND, -1, -1); + MAKE_PCI_DRIVER("vfio-pci", -1, -1);
.. except this one. This means I can drop two patches from mine series if I include this change :-) Let me resping that. Michal
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> The softlink to physfn is the way to know if the device is VF or not. So, the patch softlinks 'physfn' to the parent function. The multifunction PCI devices dont have 'physfn' softlinks. The patch adds few Virtual functions to the mock environment and changes the existing VFIO test xmls using the VFs to use the newly added VFs for their use case. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- .../hostdev-vfio-multidomain.args | 2 +- .../hostdev-vfio-multidomain.xml | 2 +- tests/qemuxml2argvdata/hostdev-vfio.args | 2 +- tests/qemuxml2argvdata/hostdev-vfio.xml | 2 +- tests/qemuxml2argvdata/net-hostdev-fail.xml | 2 +- tests/qemuxml2argvdata/net-hostdev-vfio.args | 2 +- tests/qemuxml2argvdata/net-hostdev-vfio.xml | 2 +- tests/qemuxml2xmloutdata/hostdev-vfio.xml | 2 +- tests/qemuxml2xmloutdata/net-hostdev-vfio.xml | 2 +- tests/virpcimock.c | 16 ++++++++++++++++ tests/virpcitestdata/0000-06-12.0.config | Bin 0 -> 256 bytes tests/virpcitestdata/0000-06-12.1.config | Bin 0 -> 256 bytes tests/virpcitestdata/0000-06-12.2.config | Bin 0 -> 256 bytes 13 files changed, 25 insertions(+), 9 deletions(-) create mode 100644 tests/virpcitestdata/0000-06-12.0.config create mode 100644 tests/virpcitestdata/0000-06-12.1.config create mode 100644 tests/virpcitestdata/0000-06-12.2.config diff --git a/tests/qemuxml2argvdata/hostdev-vfio-multidomain.args b/tests/qemuxml2argvdata/hostdev-vfio-multidomain.args index d098bff356..959d2a8888 100644 --- a/tests/qemuxml2argvdata/hostdev-vfio-multidomain.args +++ b/tests/qemuxml2argvdata/hostdev-vfio-multidomain.args @@ -27,5 +27,5 @@ server,nowait \ -usb \ -drive file=/dev/HostVG/QEMUGuest2,format=raw,if=none,id=drive-ide0-0-0 \ -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \ --device vfio-pci,host=55aa:20:0f.3,id=hostdev0,bus=pci.0,addr=0x3 \ +-device vfio-pci,host=0021:de:1f.1,id=hostdev0,bus=pci.0,addr=0x3 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 diff --git a/tests/qemuxml2argvdata/hostdev-vfio-multidomain.xml b/tests/qemuxml2argvdata/hostdev-vfio-multidomain.xml index 832458125b..7c34b65c7f 100644 --- a/tests/qemuxml2argvdata/hostdev-vfio-multidomain.xml +++ b/tests/qemuxml2argvdata/hostdev-vfio-multidomain.xml @@ -25,7 +25,7 @@ <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> - <address domain='0x55aa' bus='32' slot='15' function='3'/> + <address domain='0x0021' bus='222' slot='31' function='1'/> </source> </hostdev> <memballoon model='virtio'/> diff --git a/tests/qemuxml2argvdata/hostdev-vfio.args b/tests/qemuxml2argvdata/hostdev-vfio.args index b86181e267..e7456d0e49 100644 --- a/tests/qemuxml2argvdata/hostdev-vfio.args +++ b/tests/qemuxml2argvdata/hostdev-vfio.args @@ -27,5 +27,5 @@ server,nowait \ -usb \ -drive file=/dev/HostVG/QEMUGuest2,format=raw,if=none,id=drive-ide0-0-0 \ -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \ --device vfio-pci,host=06:12.5,id=hostdev0,bus=pci.0,addr=0x3 \ +-device vfio-pci,host=06:12.1,id=hostdev0,bus=pci.0,addr=0x3 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 diff --git a/tests/qemuxml2argvdata/hostdev-vfio.xml b/tests/qemuxml2argvdata/hostdev-vfio.xml index 4d96b9f732..b2e6c62e7e 100644 --- a/tests/qemuxml2argvdata/hostdev-vfio.xml +++ b/tests/qemuxml2argvdata/hostdev-vfio.xml @@ -27,7 +27,7 @@ <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> - <address domain='0x0000' bus='0x06' slot='0x12' function='0x5'/> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> </source> </hostdev> <memballoon model='virtio'/> diff --git a/tests/qemuxml2argvdata/net-hostdev-fail.xml b/tests/qemuxml2argvdata/net-hostdev-fail.xml index c815d68bd9..50b102c658 100644 --- a/tests/qemuxml2argvdata/net-hostdev-fail.xml +++ b/tests/qemuxml2argvdata/net-hostdev-fail.xml @@ -25,7 +25,7 @@ <interface type='hostdev' managed='yes'> <mac address='00:11:22:33:44:55'/> <source> - <address type='pci' domain='0x0000' bus='0x03' slot='0x07' function='0x1'/> + <address type='pci' domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> </source> <model type='virtio'/> <filterref filter='myfilter'/> diff --git a/tests/qemuxml2argvdata/net-hostdev-vfio.args b/tests/qemuxml2argvdata/net-hostdev-vfio.args index fcc8716fdf..a7fb5819ac 100644 --- a/tests/qemuxml2argvdata/net-hostdev-vfio.args +++ b/tests/qemuxml2argvdata/net-hostdev-vfio.args @@ -27,5 +27,5 @@ server,nowait \ -usb \ -drive file=/dev/HostVG/QEMUGuest1,format=raw,if=none,id=drive-ide0-0-0 \ -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \ --device vfio-pci,host=03:07.1,id=hostdev0,bus=pci.0,addr=0x3 \ +-device vfio-pci,host=06:12.1,id=hostdev0,bus=pci.0,addr=0x3 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 diff --git a/tests/qemuxml2argvdata/net-hostdev-vfio.xml b/tests/qemuxml2argvdata/net-hostdev-vfio.xml index 24034cad8b..aff681c0fb 100644 --- a/tests/qemuxml2argvdata/net-hostdev-vfio.xml +++ b/tests/qemuxml2argvdata/net-hostdev-vfio.xml @@ -26,7 +26,7 @@ <mac address='00:11:22:33:44:55'/> <driver name='vfio'/> <source> - <address type='pci' domain='0x0000' bus='0x03' slot='0x07' function='0x1'/> + <address type='pci' domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> </source> <vlan> <tag id='42'/> diff --git a/tests/qemuxml2xmloutdata/hostdev-vfio.xml b/tests/qemuxml2xmloutdata/hostdev-vfio.xml index 77bd62a129..15a845f4d1 100644 --- a/tests/qemuxml2xmloutdata/hostdev-vfio.xml +++ b/tests/qemuxml2xmloutdata/hostdev-vfio.xml @@ -32,7 +32,7 @@ <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> - <address domain='0x0000' bus='0x06' slot='0x12' function='0x5'/> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> diff --git a/tests/qemuxml2xmloutdata/net-hostdev-vfio.xml b/tests/qemuxml2xmloutdata/net-hostdev-vfio.xml index 0523cd8d3b..3f057a8249 100644 --- a/tests/qemuxml2xmloutdata/net-hostdev-vfio.xml +++ b/tests/qemuxml2xmloutdata/net-hostdev-vfio.xml @@ -31,7 +31,7 @@ <mac address='00:11:22:33:44:55'/> <driver name='vfio'/> <source> - <address type='pci' domain='0x0000' bus='0x03' slot='0x07' function='0x1'/> + <address type='pci' domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> </source> <vlan> <tag id='42'/> diff --git a/tests/virpcimock.c b/tests/virpcimock.c index deb802e860..aa1858c4a6 100644 --- a/tests/virpcimock.c +++ b/tests/virpcimock.c @@ -122,6 +122,7 @@ struct pciDevice { int device; int klass; int iommuGroup; + const char *physfn; struct pciDriver *driver; /* Driver attached. NULL if attached to no driver */ }; @@ -429,6 +430,15 @@ pci_device_new_from_stub(const struct pciDevice *data) } make_symlink(devpath, "iommu_group", tmp); + if (dev->physfn) { + if (snprintf(tmp, sizeof(tmp), + "%s/devices/%s", fakesysfspcidir, dev->physfn) < 0) { + ABORT("@tmp overflow"); + } + make_symlink(devpath, "physfn", tmp); + } + + if (pci_device_autobind(dev) < 0) ABORT("Unable to bind: %s", data->id); @@ -1003,6 +1013,12 @@ init_env(void) MAKE_PCI_DEVICE("0000:0a:01.0", 0x8086, 0x0047, .iommuGroup = 8); MAKE_PCI_DEVICE("0000:0a:02.0", 0x8286, 0x0048, .iommuGroup = 8); MAKE_PCI_DEVICE("0000:0a:03.0", 0x8386, 0x0048, .iommuGroup = 8); + MAKE_PCI_DEVICE("0000:06:12.0", 0x8086, 0x0047, .iommuGroup = 9); + MAKE_PCI_DEVICE("0000:06:12.1", 0x8086, 0x0047, .iommuGroup = 10, .physfn = "0000:06:12.0"); /* Virtual Function */ + MAKE_PCI_DEVICE("0000:06:12.2", 0x8086, 0x0047, .iommuGroup = 11, .physfn = "0000:06:12.0"); /* Virtual Function */ + MAKE_PCI_DEVICE("0021:de:1f.0", 0x8086, 0x0047, .iommuGroup = 12); + MAKE_PCI_DEVICE("0021:de:1f.1", 0x8086, 0x0047, .iommuGroup = 13, .physfn = "0021:de:1f.0"); /* Virtual Function */ + } diff --git a/tests/virpcitestdata/0000-06-12.0.config b/tests/virpcitestdata/0000-06-12.0.config new file mode 100644 index 0000000000000000000000000000000000000000..beee76534041a7020c08ae9ac03d9a349c6ea12e GIT binary patch literal 256 ycmXpOFlAt45MXi^VCG@qU|?VnU}yr8Sb;H6EeJS(Ng%<*sKv;@R0rb@MH&E<&;s)S literal 0 HcmV?d00001 diff --git a/tests/virpcitestdata/0000-06-12.1.config b/tests/virpcitestdata/0000-06-12.1.config new file mode 100644 index 0000000000000000000000000000000000000000..beee76534041a7020c08ae9ac03d9a349c6ea12e GIT binary patch literal 256 ycmXpOFlAt45MXi^VCG@qU|?VnU}yr8Sb;H6EeJS(Ng%<*sKv;@R0rb@MH&E<&;s)S literal 0 HcmV?d00001 diff --git a/tests/virpcitestdata/0000-06-12.2.config b/tests/virpcitestdata/0000-06-12.2.config new file mode 100644 index 0000000000000000000000000000000000000000..beee76534041a7020c08ae9ac03d9a349c6ea12e GIT binary patch literal 256 ycmXpOFlAt45MXi^VCG@qU|?VnU}yr8Sb;H6EeJS(Ng%<*sKv;@R0rb@MH&E<&;s)S literal 0 HcmV?d00001 -- 2.21.0
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- src/util/virprocess.h | 2 +- tests/Makefile.am | 7 +++ tests/qemuhotplugtest.c | 42 +++++++++++++- .../qemuhotplug-hostdev-pci.xml | 6 ++ .../qemuhotplug-base-live+hostdev-pci.xml | 58 +++++++++++++++++++ ...uhotplug-pseries-base-live+hostdev-pci.xml | 51 ++++++++++++++++ .../qemuhotplug-pseries-base-live.xml | 43 ++++++++++++++ tests/virprocessmock.c | 28 +++++++++ 8 files changed, 235 insertions(+), 2 deletions(-) create mode 100644 tests/qemuhotplugtestdevices/qemuhotplug-hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-base-live+hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live+hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live.xml create mode 100644 tests/virprocessmock.c diff --git a/src/util/virprocess.h b/src/util/virprocess.h index 003ba1edf4..4806c592da 100644 --- a/src/util/virprocess.h +++ b/src/util/virprocess.h @@ -75,7 +75,7 @@ int virProcessGetNamespaces(pid_t pid, int virProcessSetNamespaces(size_t nfdlist, int *fdlist); -int virProcessSetMaxMemLock(pid_t pid, unsigned long long bytes); +int virProcessSetMaxMemLock(pid_t pid, unsigned long long bytes) ATTRIBUTE_NOINLINE; int virProcessSetMaxProcesses(pid_t pid, unsigned int procs); int virProcessSetMaxFiles(pid_t pid, unsigned int files); int virProcessSetMaxCoreSize(pid_t pid, unsigned long long bytes); diff --git a/tests/Makefile.am b/tests/Makefile.am index f480e68e7d..66d249a0cc 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -210,6 +210,7 @@ test_libraries = libshunload.la \ virpcimock.la \ virnetdevmock.la \ virrandommock.la \ + virprocessmock.la \ virhostcpumock.la \ domaincapsmock.la \ virfilecachemock.la \ @@ -1182,6 +1183,12 @@ virrandommock_la_SOURCES = \ virrandommock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS) virrandommock_la_LIBADD = $(MOCKLIBS_LIBS) +virprocessmock_la_SOURCES = \ + virprocessmock.c +virprocessmock_la_CFLAGS = $(AM_CFLAGS) +virprocessmock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS) +virprocessmock_la_LIBADD = $(MOCKLIBS_LIBS) + virhostcpumock_la_SOURCES = \ virhostcpumock.c virhostcpumock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS) diff --git a/tests/qemuhotplugtest.c b/tests/qemuhotplugtest.c index 354068d748..bf95bfb4a7 100644 --- a/tests/qemuhotplugtest.c +++ b/tests/qemuhotplugtest.c @@ -28,6 +28,7 @@ #include "testutils.h" #include "testutilsqemu.h" #include "testutilsqemuschema.h" +#include "virhostdev.h" #include "virerror.h" #include "virstring.h" #include "virthread.h" @@ -79,6 +80,8 @@ qemuHotplugCreateObjects(virDomainXMLOptionPtr xmlopt, virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_IVSHMEM_PLAIN); virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_IVSHMEM_DOORBELL); virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_SCSI_DISK_WWN); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_VFIO_PCI); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_SPAPR_PCI_HOST_BRIDGE); if (qemuTestCapsCacheInsert(driver.qemuCapsCache, priv->qemuCaps) < 0) goto cleanup; @@ -130,6 +133,9 @@ testQemuHotplugAttach(virDomainObjPtr vm, case VIR_DOMAIN_DEVICE_WATCHDOG: ret = qemuDomainAttachWatchdog(&driver, vm, dev->data.watchdog); break; + case VIR_DOMAIN_DEVICE_HOSTDEV: + ret = qemuDomainAttachHostDevice(&driver, vm, dev->data.hostdev); + break; default: VIR_TEST_VERBOSE("device type '%s' cannot be attached\n", virDomainDeviceTypeToString(dev->type)); @@ -151,6 +157,7 @@ testQemuHotplugDetach(virDomainObjPtr vm, case VIR_DOMAIN_DEVICE_CHR: case VIR_DOMAIN_DEVICE_SHMEM: case VIR_DOMAIN_DEVICE_WATCHDOG: + case VIR_DOMAIN_DEVICE_HOSTDEV: ret = qemuDomainDetachDeviceLive(vm, dev, &driver, async); break; default: @@ -578,6 +585,7 @@ testQemuHotplugCpuIndividual(const void *opaque) return ret; } +#define FAKEROOTDIRTEMPLATE abs_builddir "/fakerootdir-XXXXXX" static int @@ -587,6 +595,21 @@ mymain(void) int ret = 0; struct qemuHotplugTestData data = {0}; struct testQemuHotplugCpuParams cpudata; + char *fakerootdir; + + if (VIR_STRDUP_QUIET(fakerootdir, FAKEROOTDIRTEMPLATE) < 0) { + fprintf(stderr, "Out of memory\n"); + abort(); + } + + if (!mkdtemp(fakerootdir)) { + fprintf(stderr, "Cannot create fakerootdir"); + abort(); + } + + setenv("LIBVIRT_FAKE_ROOT_DIR", fakerootdir, 1); + unsetenv("LD_PRELOAD"); + #if !WITH_YAJL fputs("libvirt not compiled with JSON support, skipping this test\n", stderr); @@ -621,6 +644,8 @@ mymain(void) if (!driver.lockManager) return EXIT_FAILURE; + driver.hostdevMgr = virHostdevManagerGetDefault(); + /* wait only 100ms for DEVICE_DELETED event */ qemuDomainRemoveDeviceWaitTime = 100; @@ -789,6 +814,15 @@ mymain(void) "human-monitor-command", HMP("OK\\r\\n"), "device_add", QMP_OK); + DO_TEST_ATTACH("base-live", "hostdev-pci", false, true, + "device_add", QMP_OK); + DO_TEST_DETACH("base-live", "hostdev-pci", false, false, + "device_del", QMP_DEVICE_DELETED("hostdev0") QMP_OK); + DO_TEST_ATTACH("pseries-base-live", "hostdev-pci", false, true, + "device_add", QMP_OK); + DO_TEST_DETACH("pseries-base-live", "hostdev-pci", false, false, + "device_del", QMP_DEVICE_DELETED("hostdev0") QMP_OK); + DO_TEST_ATTACH("base-live", "watchdog", false, true, "watchdog-set-action", QMP_OK, "device_add", QMP_OK); @@ -846,9 +880,15 @@ mymain(void) DO_TEST_CPU_INDIVIDUAL("ppc64-modern-individual", "16-22", true, true, true); DO_TEST_CPU_INDIVIDUAL("ppc64-modern-individual", "17", true, true, true); + if (getenv("LIBVIRT_SKIP_CLEANUP") == NULL) + virFileDeleteTree(fakerootdir); + VIR_FREE(fakerootdir); + qemuTestDriverFree(&driver); virObjectUnref(data.vm); return (ret == 0) ? EXIT_SUCCESS : EXIT_FAILURE; } -VIR_TEST_MAIN(mymain) +VIR_TEST_MAIN_PRELOAD(mymain, + abs_builddir "/.libs/virpcimock.so", + abs_builddir "/.libs/virprocessmock.so"); diff --git a/tests/qemuhotplugtestdevices/qemuhotplug-hostdev-pci.xml b/tests/qemuhotplugtestdevices/qemuhotplug-hostdev-pci.xml new file mode 100644 index 0000000000..6f7c99c943 --- /dev/null +++ b/tests/qemuhotplugtestdevices/qemuhotplug-hostdev-pci.xml @@ -0,0 +1,6 @@ +<hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x2'/> + </source> +</hostdev> diff --git a/tests/qemuhotplugtestdomains/qemuhotplug-base-live+hostdev-pci.xml b/tests/qemuhotplugtestdomains/qemuhotplug-base-live+hostdev-pci.xml new file mode 100644 index 0000000000..40dcc33595 --- /dev/null +++ b/tests/qemuhotplugtestdomains/qemuhotplug-base-live+hostdev-pci.xml @@ -0,0 +1,58 @@ +<domain type='kvm' id='7'> + <name>hotplug</name> + <uuid>d091ea82-29e6-2e34-3005-f02617b36e87</uuid> + <memory unit='KiB'>4194304</memory> + <currentMemory unit='KiB'>4194304</currentMemory> + <vcpu placement='static'>4</vcpu> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <features> + <acpi/> + <apic/> + <pae/> + </features> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>restart</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <controller type='usb' index='0'> + <alias name='usb'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='ide' index='0'> + <alias name='ide'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> + </controller> + <controller type='scsi' index='0' model='virtio-scsi'> + <alias name='scsi0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </controller> + <controller type='pci' index='0' model='pci-root'> + <alias name='pci'/> + </controller> + <controller type='virtio-serial' index='0'> + <alias name='virtio-serial0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> + </controller> + <input type='mouse' bus='ps2'> + <alias name='input0'/> + </input> + <input type='keyboard' bus='ps2'> + <alias name='input1'/> + </input> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x2'/> + </source> + <alias name='hostdev0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + </hostdev> + <memballoon model='none'/> + </devices> + <seclabel type='none' model='none'/> +</domain> diff --git a/tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live+hostdev-pci.xml b/tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live+hostdev-pci.xml new file mode 100644 index 0000000000..483cb5d8f0 --- /dev/null +++ b/tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live+hostdev-pci.xml @@ -0,0 +1,51 @@ +<domain type='kvm' id='7'> + <name>hotplug</name> + <uuid>d091ea82-29e6-2e34-3005-f02617b36e87</uuid> + <memory unit='KiB'>4194304</memory> + <currentMemory unit='KiB'>4194304</currentMemory> + <vcpu placement='static'>4</vcpu> + <os> + <type arch='ppc64' machine='pseries'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-ppc64</emulator> + <controller type='pci' index='0' model='pci-root'> + <model name='spapr-pci-host-bridge'/> + <target index='0'/> + <alias name='pci.0'/> + </controller> + <controller type='pci' index='1' model='pci-root'> + <model name='spapr-pci-host-bridge'/> + <target index='1'/> + <alias name='pci.1'/> + </controller> + <controller type='usb' index='0'> + <alias name='usb'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> + </controller> + <input type='keyboard' bus='usb'> + <alias name='input0'/> + <address type='usb' bus='0' port='1'/> + </input> + <input type='mouse' bus='usb'> + <alias name='input1'/> + <address type='usb' bus='0' port='2'/> + </input> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x2'/> + </source> + <alias name='hostdev0'/> + <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> + </hostdev> + <memballoon model='none'/> + <panic model='pseries'/> + </devices> + <seclabel type='none' model='none'/> +</domain> diff --git a/tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live.xml b/tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live.xml new file mode 100644 index 0000000000..1dcc35f626 --- /dev/null +++ b/tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live.xml @@ -0,0 +1,43 @@ +<domain type='kvm' id='7'> + <name>hotplug</name> + <uuid>d091ea82-29e6-2e34-3005-f02617b36e87</uuid> + <memory unit='KiB'>4194304</memory> + <currentMemory unit='KiB'>4194304</currentMemory> + <vcpu placement='static'>4</vcpu> + <os> + <type arch='ppc64' machine='pseries'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-ppc64</emulator> + <controller type='pci' index='0' model='pci-root'> + <model name='spapr-pci-host-bridge'/> + <target index='0'/> + <alias name='pci.0'/> + </controller> + <controller type='pci' index='1' model='pci-root'> + <model name='spapr-pci-host-bridge'/> + <target index='1'/> + <alias name='pci.1'/> + </controller> + <controller type='usb' index='0'> + <alias name='usb'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> + </controller> + <input type='keyboard' bus='usb'> + <alias name='input0'/> + <address type='usb' bus='0' port='1'/> + </input> + <input type='mouse' bus='usb'> + <alias name='input1'/> + <address type='usb' bus='0' port='2'/> + </input> + <memballoon model='none'/> + <panic model='pseries'/> + </devices> + <seclabel type='none' model='none'/> +</domain> diff --git a/tests/virprocessmock.c b/tests/virprocessmock.c new file mode 100644 index 0000000000..985562fee1 --- /dev/null +++ b/tests/virprocessmock.c @@ -0,0 +1,28 @@ +/* + * Copyright (C) 2018 Red Hat, Inc. + * Copyright (C) 2018 IBM Corp. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>. + * + */ + +#include <config.h> +#include "virprocess.h" + +int +virProcessSetMaxMemLock(pid_t pid ATTRIBUTE_UNUSED, unsigned long long bytes ATTRIBUTE_UNUSED) +{ + return 0; +} -- 2.21.0
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com>
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- src/util/virprocess.h | 2 +- tests/Makefile.am | 7 +++ tests/qemuhotplugtest.c | 42 +++++++++++++- .../qemuhotplug-hostdev-pci.xml | 6 ++ .../qemuhotplug-base-live+hostdev-pci.xml | 58 +++++++++++++++++++ ...uhotplug-pseries-base-live+hostdev-pci.xml | 51 ++++++++++++++++ .../qemuhotplug-pseries-base-live.xml | 43 ++++++++++++++ tests/virprocessmock.c | 28 +++++++++ 8 files changed, 235 insertions(+), 2 deletions(-) create mode 100644 tests/qemuhotplugtestdevices/qemuhotplug-hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-base-live+hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live+hostdev-pci.xml create mode 100644 tests/qemuhotplugtestdomains/qemuhotplug-pseries-base-live.xml create mode 100644 tests/virprocessmock.c
diff --git a/src/util/virprocess.h b/src/util/virprocess.h index 003ba1edf4..4806c592da 100644 --- a/src/util/virprocess.h +++ b/src/util/virprocess.h @@ -75,7 +75,7 @@ int virProcessGetNamespaces(pid_t pid, int virProcessSetNamespaces(size_t nfdlist, int *fdlist);
-int virProcessSetMaxMemLock(pid_t pid, unsigned long long bytes); +int virProcessSetMaxMemLock(pid_t pid, unsigned long long bytes) ATTRIBUTE_NOINLINE; int virProcessSetMaxProcesses(pid_t pid, unsigned int procs); int virProcessSetMaxFiles(pid_t pid, unsigned int files); int virProcessSetMaxCoreSize(pid_t pid, unsigned long long bytes); diff --git a/tests/Makefile.am b/tests/Makefile.am index f480e68e7d..66d249a0cc 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -210,6 +210,7 @@ test_libraries = libshunload.la \ virpcimock.la \ virnetdevmock.la \ virrandommock.la \ + virprocessmock.la \ virhostcpumock.la \ domaincapsmock.la \ virfilecachemock.la \ @@ -1182,6 +1183,12 @@ virrandommock_la_SOURCES = \ virrandommock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS) virrandommock_la_LIBADD = $(MOCKLIBS_LIBS)
+virprocessmock_la_SOURCES = \ + virprocessmock.c +virprocessmock_la_CFLAGS = $(AM_CFLAGS) +virprocessmock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS) +virprocessmock_la_LIBADD = $(MOCKLIBS_LIBS) + virhostcpumock_la_SOURCES = \ virhostcpumock.c virhostcpumock_la_LDFLAGS = $(MOCKLIBS_LDFLAGS) diff --git a/tests/qemuhotplugtest.c b/tests/qemuhotplugtest.c index 354068d748..bf95bfb4a7 100644 --- a/tests/qemuhotplugtest.c +++ b/tests/qemuhotplugtest.c @@ -28,6 +28,7 @@ #include "testutils.h" #include "testutilsqemu.h" #include "testutilsqemuschema.h" +#include "virhostdev.h" #include "virerror.h" #include "virstring.h" #include "virthread.h" @@ -79,6 +80,8 @@ qemuHotplugCreateObjects(virDomainXMLOptionPtr xmlopt, virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_IVSHMEM_PLAIN); virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_IVSHMEM_DOORBELL); virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_SCSI_DISK_WWN); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_VFIO_PCI); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_DEVICE_SPAPR_PCI_HOST_BRIDGE);
if (qemuTestCapsCacheInsert(driver.qemuCapsCache, priv->qemuCaps) < 0) goto cleanup; @@ -130,6 +133,9 @@ testQemuHotplugAttach(virDomainObjPtr vm, case VIR_DOMAIN_DEVICE_WATCHDOG: ret = qemuDomainAttachWatchdog(&driver, vm, dev->data.watchdog); break; + case VIR_DOMAIN_DEVICE_HOSTDEV: + ret = qemuDomainAttachHostDevice(&driver, vm, dev->data.hostdev); + break; default: VIR_TEST_VERBOSE("device type '%s' cannot be attached\n", virDomainDeviceTypeToString(dev->type)); @@ -151,6 +157,7 @@ testQemuHotplugDetach(virDomainObjPtr vm, case VIR_DOMAIN_DEVICE_CHR: case VIR_DOMAIN_DEVICE_SHMEM: case VIR_DOMAIN_DEVICE_WATCHDOG: + case VIR_DOMAIN_DEVICE_HOSTDEV: ret = qemuDomainDetachDeviceLive(vm, dev, &driver, async); break; default: @@ -578,6 +585,7 @@ testQemuHotplugCpuIndividual(const void *opaque) return ret; }
+#define FAKEROOTDIRTEMPLATE abs_builddir "/fakerootdir-XXXXXX"
static int @@ -587,6 +595,21 @@ mymain(void) int ret = 0; struct qemuHotplugTestData data = {0}; struct testQemuHotplugCpuParams cpudata; + char *fakerootdir; + + if (VIR_STRDUP_QUIET(fakerootdir, FAKEROOTDIRTEMPLATE) < 0) { + fprintf(stderr, "Out of memory\n"); + abort(); + } + + if (!mkdtemp(fakerootdir)) { + fprintf(stderr, "Cannot create fakerootdir"); + abort(); + } + + setenv("LIBVIRT_FAKE_ROOT_DIR", fakerootdir, 1); + unsetenv("LD_PRELOAD");
This unsetenv is not necessary, isn't it? Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> There are already good number of test cases with hostdevices, few have multifunction devices but none having more than one than one multifunction cards. This patch adds a case where there are two multifunction cards and two Virtual functions part of the same XML. 0001:01:00.X & 0005:09:00.X - are Multifunction PCI cards. 0000:06:12.[5|6] - are SRIOV Virtual functions The next few commits improve on automatically detecting the multifunction cards and auto-assinging the addresses appropriately. Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- .../hostdev-pci-multifunction.args | 35 ++++++++ .../hostdev-pci-multifunction.xml | 59 ++++++++++++++ tests/qemuxml2argvtest.c | 3 + .../hostdev-pci-multifunction.xml | 79 +++++++++++++++++++ tests/qemuxml2xmltest.c | 1 + 5 files changed, 177 insertions(+) create mode 100644 tests/qemuxml2argvdata/hostdev-pci-multifunction.args create mode 100644 tests/qemuxml2argvdata/hostdev-pci-multifunction.xml create mode 100644 tests/qemuxml2xmloutdata/hostdev-pci-multifunction.xml diff --git a/tests/qemuxml2argvdata/hostdev-pci-multifunction.args b/tests/qemuxml2argvdata/hostdev-pci-multifunction.args new file mode 100644 index 0000000000..7747cca442 --- /dev/null +++ b/tests/qemuxml2argvdata/hostdev-pci-multifunction.args @@ -0,0 +1,35 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/tmp/lib/domain--1-delete \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/tmp/lib/domain--1-delete/.local/share \ +XDG_CACHE_HOME=/tmp/lib/domain--1-delete/.cache \ +XDG_CONFIG_HOME=/tmp/lib/domain--1-delete/.config \ +QEMU_AUDIO_DRV=none \ +/usr/bin/qemu-system-x86_64 \ +-name delete \ +-S \ +-machine pc,accel=kvm,usb=off,dump-guest-core=off \ +-m 256 \ +-realtime mlock=off \ +-smp 4,sockets=4,cores=1,threads=1 \ +-uuid 583a8e8e-f0ce-4f53-89ab-092862148b25 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,path=/tmp/lib/domain--1-delete/monitor.sock,\ +server,nowait \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-no-acpi \ +-usb \ +-device vfio-pci,host=0005:90:01.0,id=hostdev0,bus=pci.0,addr=0x3 \ +-device vfio-pci,host=0001:01:00.1,id=hostdev1,bus=pci.0,addr=0x4 \ +-device vfio-pci,host=0001:01:00.0,id=hostdev2,bus=pci.0,addr=0x5 \ +-device vfio-pci,host=0005:90:01.2,id=hostdev3,bus=pci.0,addr=0x6 \ +-device vfio-pci,host=0005:90:01.3,id=hostdev4,bus=pci.0,addr=0x7 \ +-device vfio-pci,host=06:12.1,id=hostdev5,bus=pci.0,addr=0x8 \ +-device vfio-pci,host=06:12.2,id=hostdev6,bus=pci.0,addr=0x9 \ +-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa diff --git a/tests/qemuxml2argvdata/hostdev-pci-multifunction.xml b/tests/qemuxml2argvdata/hostdev-pci-multifunction.xml new file mode 100644 index 0000000000..06c889c64d --- /dev/null +++ b/tests/qemuxml2argvdata/hostdev-pci-multifunction.xml @@ -0,0 +1,59 @@ +<domain type='kvm'> + <name>delete</name> + <uuid>583a8e8e-f0ce-4f53-89ab-092862148b25</uuid> + <memory unit='KiB'>262144</memory> + <vcpu placement='static'>4</vcpu> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + </os> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <controller type='usb' index='0'/> + <controller type='ide' index='0'/> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0005' bus='0x90' slot='0x01' function='0x0'/> + </source> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0001' bus='0x01' slot='0x00' function='0x1'/> + </source> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0001' bus='0x01' slot='0x00' function='0x0'/> + </source> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0005' bus='0x90' slot='0x01' function='0x2'/> + </source> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0005' bus='0x90' slot='0x01' function='0x3'/> + </source> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> + </source> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x2'/> + </source> + </hostdev> + </devices> +</domain> diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index d6e6272518..5feb305f41 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -1313,6 +1313,9 @@ mymain(void) DO_TEST_FAILURE("net-hostdev-fail", QEMU_CAPS_DEVICE_VFIO_PCI); + DO_TEST("hostdev-pci-multifunction", + QEMU_CAPS_KVM, + QEMU_CAPS_DEVICE_VFIO_PCI); DO_TEST("serial-file-log", QEMU_CAPS_CHARDEV_FILE_APPEND, diff --git a/tests/qemuxml2xmloutdata/hostdev-pci-multifunction.xml b/tests/qemuxml2xmloutdata/hostdev-pci-multifunction.xml new file mode 100644 index 0000000000..52ed86e305 --- /dev/null +++ b/tests/qemuxml2xmloutdata/hostdev-pci-multifunction.xml @@ -0,0 +1,79 @@ +<domain type='kvm'> + <name>delete</name> + <uuid>583a8e8e-f0ce-4f53-89ab-092862148b25</uuid> + <memory unit='KiB'>262144</memory> + <currentMemory unit='KiB'>262144</currentMemory> + <vcpu placement='static'>4</vcpu> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <controller type='usb' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='ide' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0005' bus='0x90' slot='0x01' function='0x0'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0001' bus='0x01' slot='0x00' function='0x1'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0001' bus='0x01' slot='0x00' function='0x0'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0005' bus='0x90' slot='0x01' function='0x2'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0005' bus='0x90' slot='0x01' function='0x3'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x1'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> + </hostdev> + <hostdev mode='subsystem' type='pci' managed='yes'> + <driver name='vfio'/> + <source> + <address domain='0x0000' bus='0x06' slot='0x12' function='0x2'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> + </hostdev> + <memballoon model='virtio'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> + </memballoon> + </devices> +</domain> diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c index 09c86eda2a..b7e1b58887 100644 --- a/tests/qemuxml2xmltest.c +++ b/tests/qemuxml2xmltest.c @@ -427,6 +427,7 @@ mymain(void) DO_TEST("hostdev-usb-address", NONE); DO_TEST("hostdev-pci-address", NONE); + DO_TEST("hostdev-pci-multifunction", NONE); DO_TEST("hostdev-vfio", NONE); DO_TEST("hostdev-vfio-zpci", QEMU_CAPS_DEVICE_ZPCI, -- 2.21.0
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
From: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com>
There are already good number of test cases with hostdevices, few have multifunction devices but none having more than one than one multifunction cards.
This patch adds a case where there are two multifunction cards and two Virtual functions part of the same XML.
0001:01:00.X & 0005:09:00.X - are Multifunction PCI cards. 0000:06:12.[5|6] - are SRIOV Virtual functions
The next few commits improve on automatically detecting the multifunction cards and auto-assinging the addresses appropriately.
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.vnet.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> --- .../hostdev-pci-multifunction.args | 35 ++++++++ .../hostdev-pci-multifunction.xml | 59 ++++++++++++++ tests/qemuxml2argvtest.c | 3 + .../hostdev-pci-multifunction.xml | 79 +++++++++++++++++++ tests/qemuxml2xmltest.c | 1 + 5 files changed, 177 insertions(+) create mode 100644 tests/qemuxml2argvdata/hostdev-pci-multifunction.args create mode 100644 tests/qemuxml2argvdata/hostdev-pci-multifunction.xml create mode 100644 tests/qemuxml2xmloutdata/hostdev-pci-multifunction.xml
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal
On 7/25/19 8:09 PM, Daniel Henrique Barboza wrote:
Huh, looks like we have came across same bugs recently. Well, great minds think alike :D So I guess the resolution might be this: I'll merge my patches as soon as I get review and then you can repost your patches (it's going to be shorter series that way). Michal
participants (2)
-
Daniel Henrique Barboza -
Michal Privoznik