[PATCH v1 00/12] Bhyve driver improvements

Rebased and updated from previous patch set to address feedback: * Tried to match local convention for subjects where obvious * Split patch 01 into two patches, with updated messages * Use g_autofree to fix use after free in conf/virnetworkobj * Add missing newline in one of the tests args files * Fix failing schema tests after schema change gmake check now reports no failing tests on FreeBSD for each patch. Ryan Moeller (12): bhyve: process: remove unneeded header conf: fix use after free bhyve: process: don't bother seeking to end of log bhyve: monitor: Make bhyveMonitor a virClass bhyve: monitor: refactor register/unregister bhyve: add hooks bhyve: add reboot support bhyve: command: refactor virBhyveProcessBuildBhyveCmd bhyve: parse_command: slot,bus,func -> bus,slot,func add hostdev handling for bhyve bhyve: command: enable booting from hostdevs Allow PCI functions up to 255 for PCI ARI docs/schemas/basictypes.rng | 10 +- docs/schemas/domaincommon.rng | 30 +++ src/bhyve/bhyve_capabilities.c | 14 + src/bhyve/bhyve_capabilities.h | 1 + src/bhyve/bhyve_command.c | 241 ++++++++++++++---- src/bhyve/bhyve_driver.c | 30 +++ src/bhyve/bhyve_monitor.c | 157 ++++++++---- src/bhyve/bhyve_monitor.h | 2 + src/bhyve/bhyve_parse_command.c | 124 +++++++-- src/bhyve/bhyve_process.c | 83 ++++-- src/bhyve/bhyve_process.h | 3 + src/conf/domain_audit.c | 5 + src/conf/domain_conf.c | 131 ++++++++++ src/conf/domain_conf.h | 29 ++- src/conf/virconftypes.h | 3 + src/conf/virnetworkobj.c | 5 +- src/qemu/qemu_command.c | 2 + src/qemu/qemu_domain.c | 5 + src/qemu/qemu_hostdev.c | 1 + src/qemu/qemu_hotplug.c | 2 + src/qemu/qemu_migration.c | 1 + src/security/security_apparmor.c | 1 + src/security/security_dac.c | 28 ++ src/security/security_selinux.c | 8 + src/util/virhook.c | 15 ++ src/util/virhook.h | 11 + src/util/virpci.c | 4 +- .../bhyveargv2xml-passthru.args | 8 + .../bhyveargv2xml-passthru.xml | 26 ++ .../bhyveargv2xml-virtio-scsi.args | 9 + .../bhyveargv2xml-virtio-scsi.xml | 20 ++ tests/bhyveargv2xmltest.c | 2 + .../bhyvexml2argv-passthru.args | 11 + .../bhyvexml2argv-passthru.ldargs | 1 + .../bhyvexml2argv-passthru.xml | 22 ++ .../bhyvexml2argv-virtio-scsi.args | 9 + .../bhyvexml2argv-virtio-scsi.ldargs | 1 + .../bhyvexml2argv-virtio-scsi.xml | 21 ++ tests/bhyvexml2argvtest.c | 4 +- .../bhyvexml2xmlout-passthru.xml | 29 +++ .../bhyvexml2xmlout-virtio-scsi.xml | 23 ++ tests/bhyvexml2xmltest.c | 2 + .../qemuxml2argvdata/pci-function-invalid.xml | 2 +- 43 files changed, 983 insertions(+), 153 deletions(-) create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.args create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.xml create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.ldargs create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.args create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.ldargs create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.xml create mode 100644 tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml create mode 100644 tests/bhyvexml2xmloutdata/bhyvexml2xmlout-virtio-scsi.xml -- 2.24.1

Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 1 - 1 file changed, 1 deletion(-) diff --git a/src/bhyve/bhyve_process.c b/src/bhyve/bhyve_process.c index f05be849d4..b0b428eeb4 100644 --- a/src/bhyve/bhyve_process.c +++ b/src/bhyve/bhyve_process.c @@ -27,7 +27,6 @@ #include <sys/types.h> #include <sys/sysctl.h> #include <sys/user.h> -#include <sys/ioctl.h> #include <net/if.h> #include <net/if_tap.h> -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:13AM -0500, Ryan Moeller wrote:
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 1 - 1 file changed, 1 deletion(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/conf/virnetworkobj.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/src/conf/virnetworkobj.c b/src/conf/virnetworkobj.c index 299cdba52d..b2affaacd3 100644 --- a/src/conf/virnetworkobj.c +++ b/src/conf/virnetworkobj.c @@ -1886,7 +1886,7 @@ virNetworkObjLoadAllPorts(virNetworkObjPtr net, } while ((rc = virDirRead(dh, &de, dir)) > 0) { - char *file = NULL; + g_autofree char *file = NULL; if (!virStringStripSuffix(de->d_name, ".xml")) continue; @@ -1894,9 +1894,6 @@ virNetworkObjLoadAllPorts(virNetworkObjPtr net, file = g_strdup_printf("%s/%s.xml", dir, de->d_name); portdef = virNetworkPortDefParseFile(file); - VIR_FREE(file); - file = NULL; - if (!portdef) { VIR_WARN("Cannot parse port %s", file); continue; -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:14AM -0500, Ryan Moeller wrote:
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/conf/virnetworkobj.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

The file is opened O_APPEND. Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/bhyve/bhyve_process.c b/src/bhyve/bhyve_process.c index b0b428eeb4..a11f75a894 100644 --- a/src/bhyve/bhyve_process.c +++ b/src/bhyve/bhyve_process.c @@ -101,8 +101,6 @@ virBhyveProcessStart(virConnectPtr conn, char *devicemap = NULL; char *logfile = NULL; int logfd = -1; - off_t pos = -1; - char ebuf[1024]; virCommandPtr cmd = NULL; virCommandPtr load_cmd = NULL; bhyveConnPtr driver = conn->privateData; @@ -172,9 +170,6 @@ virBhyveProcessStart(virConnectPtr conn, /* Log generated command line */ virCommandWriteArgLog(load_cmd, logfd); - if ((pos = lseek(logfd, 0, SEEK_END)) < 0) - VIR_WARN("Unable to seek to end of logfile: %s", - virStrerror(errno, ebuf, sizeof(ebuf))); VIR_DEBUG("Loading domain '%s'", vm->def->name); if (virCommandRun(load_cmd, NULL) < 0) -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:15AM -0500, Ryan Moeller wrote:
The file is opened O_APPEND.
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 5 ----- 1 file changed, 5 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

This makes lifecycle management a bit easier thanks to ref counting, and it is closer to what the qemu driver does. Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_monitor.c | 116 +++++++++++++++++++++++++------------- 1 file changed, 76 insertions(+), 40 deletions(-) diff --git a/src/bhyve/bhyve_monitor.c b/src/bhyve/bhyve_monitor.c index 0e55e19772..58d40e1f70 100644 --- a/src/bhyve/bhyve_monitor.c +++ b/src/bhyve/bhyve_monitor.c @@ -32,24 +32,51 @@ #include "virerror.h" #include "virfile.h" #include "virlog.h" +#include "virobject.h" #define VIR_FROM_THIS VIR_FROM_BHYVE VIR_LOG_INIT("bhyve.bhyve_monitor"); struct _bhyveMonitor { + virObject parent; + int kq; int watch; bhyveConnPtr driver; + virDomainObjPtr vm; }; +static virClassPtr bhyveMonitorClass; + +static void +bhyveMonitorDispose(void *obj) +{ + bhyveMonitorPtr mon = obj; + + VIR_FORCE_CLOSE(mon->kq); + virObjectUnref(mon->vm); +} + +static int +bhyveMonitorOnceInit(void) +{ + if (!VIR_CLASS_NEW(bhyveMonitor, virClassForObject())) + return -1; + + return 0; +} + +VIR_ONCE_GLOBAL_INIT(bhyveMonitor); + static void bhyveMonitorIO(int watch, int kq, int events G_GNUC_UNUSED, void *opaque) { const struct timespec zerowait = { 0, 0 }; - virDomainObjPtr vm = opaque; - bhyveDomainObjPrivatePtr priv = vm->privateData; - bhyveMonitorPtr mon = priv->mon; + bhyveMonitorPtr mon = opaque; + virDomainObjPtr vm = mon->vm; + bhyveConnPtr driver = mon->driver; + const char *name; struct kevent kev; int rc, status; @@ -82,60 +109,49 @@ bhyveMonitorIO(int watch, int kq, int events G_GNUC_UNUSED, void *opaque) return; } + name = vm->def->name; status = kev.data; if (WIFSIGNALED(status) && WCOREDUMP(status)) { virReportError(VIR_ERR_INTERNAL_ERROR, _("Guest %s got signal %d and crashed"), - vm->def->name, - WTERMSIG(status)); - virBhyveProcessStop(mon->driver, vm, - VIR_DOMAIN_SHUTOFF_CRASHED); + name, WTERMSIG(status)); + virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_CRASHED); } else if (WIFEXITED(status)) { if (WEXITSTATUS(status) == 0) { /* 0 - reboot */ /* TODO: Implementing reboot is a little more complicated. */ - VIR_INFO("Guest %s rebooted; destroying domain.", - vm->def->name); - virBhyveProcessStop(mon->driver, vm, - VIR_DOMAIN_SHUTOFF_SHUTDOWN); + VIR_INFO("Guest %s rebooted; destroying domain.", name); + virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_SHUTDOWN); } else if (WEXITSTATUS(status) < 3) { /* 1 - shutdown, 2 - halt, 3 - triple fault. others - error */ - VIR_INFO("Guest %s shut itself down; destroying domain.", - vm->def->name); - virBhyveProcessStop(mon->driver, vm, - VIR_DOMAIN_SHUTOFF_SHUTDOWN); + VIR_INFO("Guest %s shut itself down; destroying domain.", name); + virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_SHUTDOWN); } else { VIR_INFO("Guest %s had an error and exited with status %d; destroying domain.", - vm->def->name, WEXITSTATUS(status)); - virBhyveProcessStop(mon->driver, vm, - VIR_DOMAIN_SHUTOFF_UNKNOWN); + name, WEXITSTATUS(status)); + virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_UNKNOWN); } } } } -static void -bhyveMonitorRelease(void *opaque) -{ - virDomainObjPtr vm = opaque; - bhyveDomainObjPrivatePtr priv = vm->privateData; - bhyveMonitorPtr mon = priv->mon; - - VIR_FORCE_CLOSE(mon->kq); - VIR_FREE(mon); -} - -bhyveMonitorPtr -bhyveMonitorOpen(virDomainObjPtr vm, bhyveConnPtr driver) +static bhyveMonitorPtr +bhyveMonitorOpenImpl(virDomainObjPtr vm, bhyveConnPtr driver) { - bhyveMonitorPtr mon = NULL; + bhyveMonitorPtr mon; struct kevent kev; - if (VIR_ALLOC(mon) < 0) + if (bhyveMonitorInitialize() < 0) + return NULL; + + if (!(mon = virObjectNew(bhyveMonitorClass))) return NULL; mon->driver = driver; + virObjectRef(vm); + mon->vm = vm; + mon->kq = kqueue(); if (mon->kq < 0) { virReportError(VIR_ERR_SYSTEM_ERROR, "%s", @@ -150,14 +166,17 @@ bhyveMonitorOpen(virDomainObjPtr vm, bhyveConnPtr driver) goto cleanup; } + virObjectRef(mon); mon->watch = virEventAddHandle(mon->kq, VIR_EVENT_HANDLE_READABLE | VIR_EVENT_HANDLE_ERROR | VIR_EVENT_HANDLE_HANGUP, bhyveMonitorIO, - vm, - bhyveMonitorRelease); + mon, + virObjectFreeCallback); if (mon->watch < 0) { + VIR_DEBUG("failed to add event handle for mon %p", mon); + virObjectUnref(mon); virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("unable to register monitor events")); goto cleanup; @@ -166,18 +185,35 @@ bhyveMonitorOpen(virDomainObjPtr vm, bhyveConnPtr driver) return mon; cleanup: - bhyveMonitorRelease(mon); + bhyveMonitorClose(mon); return NULL; } +bhyveMonitorPtr +bhyveMonitorOpen(virDomainObjPtr vm, bhyveConnPtr driver) +{ + bhyveMonitorPtr mon; + + virObjectRef(vm); + mon = bhyveMonitorOpenImpl(vm, driver); + virObjectUnref(vm); + + return mon; +} + void bhyveMonitorClose(bhyveMonitorPtr mon) { if (mon == NULL) return; - if (mon->watch > 0) - virEventRemoveHandle(mon->watch); - else - bhyveMonitorRelease(mon); + VIR_DEBUG("cleaning up bhyveMonitor %p", mon); + + if (mon->watch < 0) + return; + + virEventRemoveHandle(mon->watch); + mon->watch = -1; + + virObjectUnref(mon); } -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:16AM -0500, Ryan Moeller wrote:
This makes lifecycle management a bit easier thanks to ref counting, and it is closer to what the qemu driver does.
FWIW, if you want todo any more conversions in the future, note that we're intending to replace virObject with GObject. eg as illustrated in: commit 16121a88a7ef933220bcf9eff6575367278a06f7 Author: Daniel P. Berrangé <berrange@redhat.com> Date: Thu Sep 19 15:38:03 2019 +0100 util: convert virIdentity class to use GObject
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_monitor.c | 116 +++++++++++++++++++++++++------------- 1 file changed, 76 insertions(+), 40 deletions(-)
None the less, I'm fine taking this conversion to virObject as is, since it is better than what currently exists. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Pull the code for registering and unregistering a bhyve monitor object into separate functions to improve code clarity. Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_monitor.c | 50 ++++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/src/bhyve/bhyve_monitor.c b/src/bhyve/bhyve_monitor.c index 58d40e1f70..a1b1a21a6f 100644 --- a/src/bhyve/bhyve_monitor.c +++ b/src/bhyve/bhyve_monitor.c @@ -69,6 +69,37 @@ bhyveMonitorOnceInit(void) VIR_ONCE_GLOBAL_INIT(bhyveMonitor); +static void bhyveMonitorIO(int, int, int, void *); + +static bool +bhyveMonitorRegister(bhyveMonitorPtr mon) +{ + virObjectRef(mon); + mon->watch = virEventAddHandle(mon->kq, + VIR_EVENT_HANDLE_READABLE | + VIR_EVENT_HANDLE_ERROR | + VIR_EVENT_HANDLE_HANGUP, + bhyveMonitorIO, + mon, + virObjectFreeCallback); + if (mon->watch < 0) { + VIR_DEBUG("failed to add event handle for mon %p", mon); + virObjectUnref(mon); + return false; + } + return true; +} + +static void +bhyveMonitorUnregister(bhyveMonitorPtr mon) +{ + if (mon->watch < 0) + return; + + virEventRemoveHandle(mon->watch); + mon->watch = -1; +} + static void bhyveMonitorIO(int watch, int kq, int events G_GNUC_UNUSED, void *opaque) { @@ -166,17 +197,7 @@ bhyveMonitorOpenImpl(virDomainObjPtr vm, bhyveConnPtr driver) goto cleanup; } - virObjectRef(mon); - mon->watch = virEventAddHandle(mon->kq, - VIR_EVENT_HANDLE_READABLE | - VIR_EVENT_HANDLE_ERROR | - VIR_EVENT_HANDLE_HANGUP, - bhyveMonitorIO, - mon, - virObjectFreeCallback); - if (mon->watch < 0) { - VIR_DEBUG("failed to add event handle for mon %p", mon); - virObjectUnref(mon); + if (!bhyveMonitorRegister(mon)) { virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("unable to register monitor events")); goto cleanup; @@ -209,11 +230,6 @@ bhyveMonitorClose(bhyveMonitorPtr mon) VIR_DEBUG("cleaning up bhyveMonitor %p", mon); - if (mon->watch < 0) - return; - - virEventRemoveHandle(mon->watch); - mon->watch = -1; - + bhyveMonitorUnregister(mon); virObjectUnref(mon); } -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:17AM -0500, Ryan Moeller wrote:
Pull the code for registering and unregistering a bhyve monitor object into separate functions to improve code clarity.
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_monitor.c | 50 ++++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 17 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 33 +++++++++++++++++++++++++++++++++ src/util/virhook.c | 15 +++++++++++++++ src/util/virhook.h | 11 +++++++++++ 3 files changed, 59 insertions(+) diff --git a/src/bhyve/bhyve_process.c b/src/bhyve/bhyve_process.c index a11f75a894..45162503d4 100644 --- a/src/bhyve/bhyve_process.c +++ b/src/bhyve/bhyve_process.c @@ -37,6 +37,7 @@ #include "bhyve_process.h" #include "datatypes.h" #include "virerror.h" +#include "virhook.h" #include "virlog.h" #include "virfile.h" #include "viralloc.h" @@ -91,6 +92,24 @@ virBhyveFormatDevMapFile(const char *vm_name, char **fn_out) *fn_out = g_strdup_printf("%s/grub_bhyve-%s-device.map", BHYVE_STATE_DIR, vm_name); } +static int +bhyveProcessStartHook(virDomainObjPtr vm, virHookBhyveOpType op) +{ + if (!virHookPresent(VIR_HOOK_DRIVER_BHYVE)) + return 0; + + return virHookCall(VIR_HOOK_DRIVER_BHYVE, vm->def->name, op, + VIR_HOOK_SUBOP_BEGIN, NULL, NULL, NULL); +} + +static void +bhyveProcessStopHook(virDomainObjPtr vm, virHookBhyveOpType op) +{ + if (virHookPresent(VIR_HOOK_DRIVER_BHYVE)) + virHookCall(VIR_HOOK_DRIVER_BHYVE, vm->def->name, op, + VIR_HOOK_SUBOP_END, NULL, NULL, NULL); +} + int virBhyveProcessStart(virConnectPtr conn, virDomainObjPtr vm, @@ -135,6 +154,10 @@ virBhyveProcessStart(virConnectPtr conn, if (bhyveDomainAssignAddresses(vm->def, NULL) < 0) goto cleanup; + /* Run an early hook to setup missing devices. */ + if (bhyveProcessStartHook(vm, VIR_HOOK_BHYVE_OP_PREPARE) < 0) + goto cleanup; + /* Call bhyve to start the VM */ if (!(cmd = virBhyveProcessBuildBhyveCmd(driver, vm->def, false))) goto cleanup; @@ -176,6 +199,9 @@ virBhyveProcessStart(virConnectPtr conn, goto cleanup; } + if (bhyveProcessStartHook(vm, VIR_HOOK_BHYVE_OP_START) < 0) + goto cleanup; + /* Now we can start the domain */ VIR_DEBUG("Starting domain '%s'", vm->def->name); if (virCommandRun(cmd, NULL) < 0) @@ -200,6 +226,9 @@ virBhyveProcessStart(virConnectPtr conn, BHYVE_STATE_DIR) < 0) goto cleanup; + if (bhyveProcessStartHook(vm, VIR_HOOK_BHYVE_OP_STARTED) < 0) + goto cleanup; + ret = 0; cleanup: @@ -263,6 +292,8 @@ virBhyveProcessStop(bhyveConnPtr driver, if ((priv != NULL) && (priv->mon != NULL)) bhyveMonitorClose(priv->mon); + bhyveProcessStopHook(vm, VIR_HOOK_BHYVE_OP_STOPPED); + /* Cleanup network interfaces */ bhyveNetCleanup(vm); @@ -284,6 +315,8 @@ virBhyveProcessStop(bhyveConnPtr driver, vm->pid = -1; vm->def->id = -1; + bhyveProcessStopHook(vm, VIR_HOOK_BHYVE_OP_RELEASE); + cleanup: virCommandFree(cmd); diff --git a/src/util/virhook.c b/src/util/virhook.c index 5bdacdd79f..e499841f66 100644 --- a/src/util/virhook.c +++ b/src/util/virhook.c @@ -47,6 +47,7 @@ VIR_ENUM_DECL(virHookQemuOp); VIR_ENUM_DECL(virHookLxcOp); VIR_ENUM_DECL(virHookNetworkOp); VIR_ENUM_DECL(virHookLibxlOp); +VIR_ENUM_DECL(virHookBhyveOp); VIR_ENUM_IMPL(virHookDriver, VIR_HOOK_DRIVER_LAST, @@ -55,6 +56,7 @@ VIR_ENUM_IMPL(virHookDriver, "lxc", "network", "libxl", + "bhyve", ); VIR_ENUM_IMPL(virHookDaemonOp, @@ -115,6 +117,15 @@ VIR_ENUM_IMPL(virHookLibxlOp, "reconnect", ); +VIR_ENUM_IMPL(virHookBhyveOp, + VIR_HOOK_BHYVE_OP_LAST, + "start", + "stopped", + "prepare", + "release", + "started", +); + static int virHooksFound = -1; /** @@ -283,6 +294,10 @@ virHookCall(int driver, break; case VIR_HOOK_DRIVER_NETWORK: opstr = virHookNetworkOpTypeToString(op); + break; + case VIR_HOOK_DRIVER_BHYVE: + opstr = virHookBhyveOpTypeToString(op); + break; } if (opstr == NULL) { virReportError(VIR_ERR_INTERNAL_ERROR, diff --git a/src/util/virhook.h b/src/util/virhook.h index f91cb87b10..d8237c837e 100644 --- a/src/util/virhook.h +++ b/src/util/virhook.h @@ -29,6 +29,7 @@ typedef enum { VIR_HOOK_DRIVER_LXC, /* LXC domains related events */ VIR_HOOK_DRIVER_NETWORK, /* network related events */ VIR_HOOK_DRIVER_LIBXL, /* Xen libxl domains related events */ + VIR_HOOK_DRIVER_BHYVE, /* Bhyve domains related events */ VIR_HOOK_DRIVER_LAST, } virHookDriverType; @@ -97,6 +98,16 @@ typedef enum { VIR_HOOK_LIBXL_OP_LAST, } virHookLibxlOpType; +typedef enum { + VIR_HOOK_BHYVE_OP_START, /* domain is about to start */ + VIR_HOOK_BHYVE_OP_STOPPED, /* domain has stopped */ + VIR_HOOK_BHYVE_OP_PREPARE, /* domain startup initiated */ + VIR_HOOK_BHYVE_OP_RELEASE, /* domain destruction is over */ + VIR_HOOK_BHYVE_OP_STARTED, /* domain has started */ + + VIR_HOOK_BHYVE_OP_LAST, +} virHookBhyveOpType; + int virHookInitialize(void); int virHookPresent(int driver); -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:18AM -0500, Ryan Moeller wrote:
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 33 +++++++++++++++++++++++++++++++++ src/util/virhook.c | 15 +++++++++++++++ src/util/virhook.h | 11 +++++++++++ 3 files changed, 59 insertions(+)
You'll also want to update the docs in docs/hooks.html.in. I'll take this patch as is though, so can you send the docs as a followup patch afterwards. Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Will do, thanks! On Mon, Feb 24, 2020 at 12:01 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Feb 24, 2020 at 01:46:18AM -0500, Ryan Moeller wrote:
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_process.c | 33 +++++++++++++++++++++++++++++++++ src/util/virhook.c | 15 +++++++++++++++ src/util/virhook.h | 11 +++++++++++ 3 files changed, 59 insertions(+)
You'll also want to update the docs in docs/hooks.html.in. I'll take this patch as is though, so can you send the docs as a followup patch afterwards.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- Ryan Moeller iXsystems, Inc. OS Developer Email: ryan@iXsystems.com

Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_driver.c | 30 ++++++++++++++++++++++ src/bhyve/bhyve_monitor.c | 19 +++++++++----- src/bhyve/bhyve_monitor.h | 2 ++ src/bhyve/bhyve_process.c | 52 ++++++++++++++++++++++++++++----------- src/bhyve/bhyve_process.h | 3 +++ 5 files changed, 85 insertions(+), 21 deletions(-) diff --git a/src/bhyve/bhyve_driver.c b/src/bhyve/bhyve_driver.c index 625dc0ec22..05ffc2f050 100644 --- a/src/bhyve/bhyve_driver.c +++ b/src/bhyve/bhyve_driver.c @@ -1013,6 +1013,35 @@ bhyveDomainShutdown(virDomainPtr dom) return bhyveDomainShutdownFlags(dom, 0); } +static int +bhyveDomainReboot(virDomainPtr dom, unsigned int flags) +{ + virConnectPtr conn = dom->conn; + virDomainObjPtr vm; + bhyveDomainObjPrivatePtr priv; + int ret = -1; + + virCheckFlags(VIR_DOMAIN_REBOOT_ACPI_POWER_BTN, -1); + + if (!(vm = bhyveDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainRebootEnsureACL(conn, vm->def, flags) < 0) + goto cleanup; + + if (virDomainObjCheckActive(vm) < 0) + goto cleanup; + + priv = vm->privateData; + bhyveMonitorSetReboot(priv->mon); + + ret = virBhyveProcessShutdown(vm); + + cleanup: + virDomainObjEndAPI(&vm); + return ret; +} + static int bhyveDomainOpenConsole(virDomainPtr dom, const char *dev_name G_GNUC_UNUSED, @@ -1657,6 +1686,7 @@ static virHypervisorDriver bhyveHypervisorDriver = { .domainDestroyFlags = bhyveDomainDestroyFlags, /* 5.6.0 */ .domainShutdown = bhyveDomainShutdown, /* 1.3.3 */ .domainShutdownFlags = bhyveDomainShutdownFlags, /* 5.6.0 */ + .domainReboot = bhyveDomainReboot, /* TBD */ .domainLookupByUUID = bhyveDomainLookupByUUID, /* 1.2.2 */ .domainLookupByName = bhyveDomainLookupByName, /* 1.2.2 */ .domainLookupByID = bhyveDomainLookupByID, /* 1.2.3 */ diff --git a/src/bhyve/bhyve_monitor.c b/src/bhyve/bhyve_monitor.c index a1b1a21a6f..e5cd39a086 100644 --- a/src/bhyve/bhyve_monitor.c +++ b/src/bhyve/bhyve_monitor.c @@ -41,10 +41,11 @@ VIR_LOG_INIT("bhyve.bhyve_monitor"); struct _bhyveMonitor { virObject parent; - int kq; - int watch; bhyveConnPtr driver; virDomainObjPtr vm; + int kq; + int watch; + bool reboot; }; static virClassPtr bhyveMonitorClass; @@ -100,6 +101,12 @@ bhyveMonitorUnregister(bhyveMonitorPtr mon) mon->watch = -1; } +void +bhyveMonitorSetReboot(bhyveMonitorPtr mon) +{ + mon->reboot = true; +} + static void bhyveMonitorIO(int watch, int kq, int events G_GNUC_UNUSED, void *opaque) { @@ -148,11 +155,10 @@ bhyveMonitorIO(int watch, int kq, int events G_GNUC_UNUSED, void *opaque) name, WTERMSIG(status)); virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_CRASHED); } else if (WIFEXITED(status)) { - if (WEXITSTATUS(status) == 0) { + if (WEXITSTATUS(status) == 0 || mon->reboot) { /* 0 - reboot */ - /* TODO: Implementing reboot is a little more complicated. */ - VIR_INFO("Guest %s rebooted; destroying domain.", name); - virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_SHUTDOWN); + VIR_INFO("Guest %s rebooted; restarting domain.", name); + virBhyveProcessRestart(driver, vm); } else if (WEXITSTATUS(status) < 3) { /* 1 - shutdown, 2 - halt, 3 - triple fault. others - error */ VIR_INFO("Guest %s shut itself down; destroying domain.", name); @@ -179,6 +185,7 @@ bhyveMonitorOpenImpl(virDomainObjPtr vm, bhyveConnPtr driver) return NULL; mon->driver = driver; + mon->reboot = false; virObjectRef(vm); mon->vm = vm; diff --git a/src/bhyve/bhyve_monitor.h b/src/bhyve/bhyve_monitor.h index 226866e6d9..175cc87192 100644 --- a/src/bhyve/bhyve_monitor.h +++ b/src/bhyve/bhyve_monitor.h @@ -29,3 +29,5 @@ typedef bhyveMonitor *bhyveMonitorPtr; bhyveMonitorPtr bhyveMonitorOpen(virDomainObjPtr vm, bhyveConnPtr driver); void bhyveMonitorClose(bhyveMonitorPtr mon); + +void bhyveMonitorSetReboot(bhyveMonitorPtr mon); diff --git a/src/bhyve/bhyve_process.c b/src/bhyve/bhyve_process.c index 45162503d4..060018bc70 100644 --- a/src/bhyve/bhyve_process.c +++ b/src/bhyve/bhyve_process.c @@ -110,11 +110,10 @@ bhyveProcessStopHook(virDomainObjPtr vm, virHookBhyveOpType op) VIR_HOOK_SUBOP_END, NULL, NULL, NULL); } -int -virBhyveProcessStart(virConnectPtr conn, - virDomainObjPtr vm, - virDomainRunningReason reason, - unsigned int flags) +static int +virBhyveProcessStartImpl(bhyveConnPtr driver, + virDomainObjPtr vm, + virDomainRunningReason reason) { char *devmap_file = NULL; char *devicemap = NULL; @@ -122,7 +121,6 @@ virBhyveProcessStart(virConnectPtr conn, int logfd = -1; virCommandPtr cmd = NULL; virCommandPtr load_cmd = NULL; - bhyveConnPtr driver = conn->privateData; bhyveDomainObjPrivatePtr priv = vm->privateData; int ret = -1, rc; @@ -154,10 +152,6 @@ virBhyveProcessStart(virConnectPtr conn, if (bhyveDomainAssignAddresses(vm->def, NULL) < 0) goto cleanup; - /* Run an early hook to setup missing devices. */ - if (bhyveProcessStartHook(vm, VIR_HOOK_BHYVE_OP_PREPARE) < 0) - goto cleanup; - /* Call bhyve to start the VM */ if (!(cmd = virBhyveProcessBuildBhyveCmd(driver, vm->def, false))) goto cleanup; @@ -213,11 +207,6 @@ virBhyveProcessStart(virConnectPtr conn, goto cleanup; } - if (flags & VIR_BHYVE_PROCESS_START_AUTODESTROY && - virCloseCallbacksSet(driver->closeCallbacks, vm, - conn, bhyveProcessAutoDestroy) < 0) - goto cleanup; - vm->def->id = vm->pid; virDomainObjSetState(vm, VIR_DOMAIN_RUNNING, reason); priv->mon = bhyveMonitorOpen(vm, driver); @@ -262,6 +251,26 @@ virBhyveProcessStart(virConnectPtr conn, return ret; } +int +virBhyveProcessStart(virConnectPtr conn, + virDomainObjPtr vm, + virDomainRunningReason reason, + unsigned int flags) +{ + bhyveConnPtr driver = conn->privateData; + + /* Run an early hook to setup missing devices. */ + if (bhyveProcessStartHook(vm, VIR_HOOK_BHYVE_OP_PREPARE) < 0) + return -1; + + if (flags & VIR_BHYVE_PROCESS_START_AUTODESTROY && + virCloseCallbacksSet(driver->closeCallbacks, vm, + conn, bhyveProcessAutoDestroy) < 0) + return -1; + + return virBhyveProcessStartImpl(driver, vm, reason); +} + int virBhyveProcessStop(bhyveConnPtr driver, virDomainObjPtr vm, @@ -349,6 +358,19 @@ virBhyveProcessShutdown(virDomainObjPtr vm) return 0; } +int +virBhyveProcessRestart(bhyveConnPtr driver, + virDomainObjPtr vm) +{ + if (virBhyveProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_SHUTDOWN) < 0) + return -1; + + if (virBhyveProcessStartImpl(driver, vm, VIR_DOMAIN_RUNNING_BOOTED) < 0) + return -1; + + return 0; +} + int virBhyveGetDomainTotalCpuStats(virDomainObjPtr vm, unsigned long long *cpustats) diff --git a/src/bhyve/bhyve_process.h b/src/bhyve/bhyve_process.h index 8419e44faa..d7b4e0bd4e 100644 --- a/src/bhyve/bhyve_process.h +++ b/src/bhyve/bhyve_process.h @@ -32,6 +32,9 @@ int virBhyveProcessStop(bhyveConnPtr driver, virDomainObjPtr vm, virDomainShutoffReason reason); +int virBhyveProcessRestart(bhyveConnPtr driver, + virDomainObjPtr vm); + int virBhyveProcessShutdown(virDomainObjPtr vm); int virBhyveGetDomainTotalCpuStats(virDomainObjPtr vm, -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:19AM -0500, Ryan Moeller wrote:
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_driver.c | 30 ++++++++++++++++++++++ src/bhyve/bhyve_monitor.c | 19 +++++++++----- src/bhyve/bhyve_monitor.h | 2 ++ src/bhyve/bhyve_process.c | 52 ++++++++++++++++++++++++++++----------- src/bhyve/bhyve_process.h | 3 +++ 5 files changed, 85 insertions(+), 21 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Reduce the complexity by isolating loop bodies in separate functions. Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_command.c | 115 ++++++++++++++++++++++---------------- 1 file changed, 67 insertions(+), 48 deletions(-) diff --git a/src/bhyve/bhyve_command.c b/src/bhyve/bhyve_command.c index 8eb99cc876..5b1d80083a 100644 --- a/src/bhyve/bhyve_command.c +++ b/src/bhyve/bhyve_command.c @@ -40,9 +40,9 @@ VIR_LOG_INIT("bhyve.bhyve_command"); static int -bhyveBuildNetArgStr(bhyveConnPtr driver, - const virDomainDef *def, +bhyveBuildNetArgStr(const virDomainDef *def, virDomainNetDefPtr net, + bhyveConnPtr driver, virCommandPtr cmd, bool dryRun) { @@ -307,6 +307,61 @@ bhyveBuildVirtIODiskArgStr(const virDomainDef *def G_GNUC_UNUSED, return 0; } +static int +bhyveBuildDiskArgStr(const virDomainDef *def, + virDomainDiskDefPtr disk, + virCommandPtr cmd) +{ + switch (disk->bus) { + case VIR_DOMAIN_DISK_BUS_SATA: + /* Handled by bhyveBuildAHCIControllerArgStr() */ + break; + case VIR_DOMAIN_DISK_BUS_VIRTIO: + if (bhyveBuildVirtIODiskArgStr(def, disk, cmd) < 0) + return -1; + break; + default: + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported disk device")); + return -1; + } + return 0; +} + +static int +bhyveBuildControllerArgStr(const virDomainDef *def, + virDomainControllerDefPtr controller, + bhyveConnPtr driver, + virCommandPtr cmd, + unsigned *nusbcontrollers) +{ + switch (controller->type) { + case VIR_DOMAIN_CONTROLLER_TYPE_PCI: + if (controller->model != VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported PCI controller model: " + "only PCI root supported")); + return -1; + } + break; + case VIR_DOMAIN_CONTROLLER_TYPE_SATA: + if (bhyveBuildAHCIControllerArgStr(def, controller, driver, cmd) < 0) + return -1; + break; + case VIR_DOMAIN_CONTROLLER_TYPE_USB: + if (++*nusbcontrollers > 1) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("only single USB controller is supported")); + return -1; + } + + if (bhyveBuildUSBControllerArgStr(def, controller, cmd) < 0) + return -1; + break; + } + return 0; +} + static int bhyveBuildLPCArgStr(const virDomainDef *def G_GNUC_UNUSED, virCommandPtr cmd) @@ -428,8 +483,8 @@ bhyveBuildGraphicsArgStr(const virDomainDef *def, } virCommandPtr -virBhyveProcessBuildBhyveCmd(bhyveConnPtr driver, - virDomainDefPtr def, bool dryRun) +virBhyveProcessBuildBhyveCmd(bhyveConnPtr driver, virDomainDefPtr def, + bool dryRun) { /* * /usr/sbin/bhyve -c 2 -m 256 -AI -H -P \ @@ -439,11 +494,10 @@ virBhyveProcessBuildBhyveCmd(bhyveConnPtr driver, * -S 31,uart,stdio \ * vm0 */ - size_t i; - int nusbcontrollers = 0; - unsigned int nvcpus = virDomainDefGetVcpus(def); - virCommandPtr cmd = virCommandNew(BHYVE); + size_t i; + unsigned nusbcontrollers = 0; + unsigned nvcpus = virDomainDefGetVcpus(def); /* CPUs */ virCommandAddArg(cmd, "-c"); @@ -547,52 +601,17 @@ virBhyveProcessBuildBhyveCmd(bhyveConnPtr driver, /* Devices */ for (i = 0; i < def->ncontrollers; i++) { - virDomainControllerDefPtr controller = def->controllers[i]; - switch (controller->type) { - case VIR_DOMAIN_CONTROLLER_TYPE_PCI: - if (controller->model != VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT) { - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - "%s", _("unsupported PCI controller model: only PCI root supported")); - goto error; - } - break; - case VIR_DOMAIN_CONTROLLER_TYPE_SATA: - if (bhyveBuildAHCIControllerArgStr(def, controller, driver, cmd) < 0) - goto error; - break; - case VIR_DOMAIN_CONTROLLER_TYPE_USB: - if (++nusbcontrollers > 1) { - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - "%s", _("only single USB controller is supported")); - goto error; - } - - if (bhyveBuildUSBControllerArgStr(def, controller, cmd) < 0) - goto error; - break; - } + if (bhyveBuildControllerArgStr(def, def->controllers[i], driver, cmd, + &nusbcontrollers) < 0) + goto error; } for (i = 0; i < def->nnets; i++) { - virDomainNetDefPtr net = def->nets[i]; - if (bhyveBuildNetArgStr(driver, def, net, cmd, dryRun) < 0) + if (bhyveBuildNetArgStr(def, def->nets[i], driver, cmd, dryRun) < 0) goto error; } for (i = 0; i < def->ndisks; i++) { - virDomainDiskDefPtr disk = def->disks[i]; - - switch (disk->bus) { - case VIR_DOMAIN_DISK_BUS_SATA: - /* Handled by bhyveBuildAHCIControllerArgStr() */ - break; - case VIR_DOMAIN_DISK_BUS_VIRTIO: - if (bhyveBuildVirtIODiskArgStr(def, disk, cmd) < 0) - goto error; - break; - default: - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", - _("unsupported disk device")); + if (bhyveBuildDiskArgStr(def, def->disks[i], cmd) < 0) goto error; - } } if (def->ngraphics && def->nvideos) { -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:20AM -0500, Ryan Moeller wrote:
Reduce the complexity by isolating loop bodies in separate functions.
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_command.c | 115 ++++++++++++++++++++++---------------- 1 file changed, 67 insertions(+), 48 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

This *is* a no-op, but there was a period of sickening dread while auditing to be sure that no actual confusion between bus and slot had occurred. I hope to avoid that by following the conventional order. Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_parse_command.c | 34 ++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/src/bhyve/bhyve_parse_command.c b/src/bhyve/bhyve_parse_command.c index dd6a626ff4..76423730d9 100644 --- a/src/bhyve/bhyve_parse_command.c +++ b/src/bhyve/bhyve_parse_command.c @@ -352,8 +352,8 @@ bhyveParseBhyveLPCArg(virDomainDefPtr def, static int bhyveParsePCISlot(const char *slotdef, - unsigned *pcislot, unsigned *bus, + unsigned *slot, unsigned *function) { /* slot[:function] | bus:slot:function */ @@ -385,7 +385,7 @@ bhyveParsePCISlot(const char *slotdef, } *bus = 0; - *pcislot = 0; + *slot = 0; *function = 0; switch (i + 1) { @@ -393,12 +393,12 @@ bhyveParsePCISlot(const char *slotdef, /* pcislot[:function] */ *function = values[1]; case 1: - *pcislot = values[0]; + *slot = values[0]; break; case 3: /* bus:pcislot:function */ *bus = values[0]; - *pcislot = values[1]; + *slot = values[1]; *function = values[2]; break; } @@ -409,8 +409,8 @@ bhyveParsePCISlot(const char *slotdef, static int bhyveParsePCIDisk(virDomainDefPtr def, unsigned caps G_GNUC_UNUSED, - unsigned pcislot, unsigned pcibus, + unsigned pcislot, unsigned function, int bus, int device, @@ -430,8 +430,8 @@ bhyveParsePCIDisk(virDomainDefPtr def, disk->device = device; disk->info.type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; - disk->info.addr.pci.slot = pcislot; disk->info.addr.pci.bus = pcibus; + disk->info.addr.pci.slot = pcislot; disk->info.addr.pci.function = function; if (STRPREFIX(config, "/dev/")) @@ -480,8 +480,8 @@ static int bhyveParsePCINet(virDomainDefPtr def, virDomainXMLOptionPtr xmlopt, unsigned caps G_GNUC_UNUSED, - unsigned pcislot, - unsigned pcibus, + unsigned bus, + unsigned slot, unsigned function, int model, const char *config) @@ -503,8 +503,8 @@ bhyveParsePCINet(virDomainDefPtr def, net->model = model; net->info.type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; - net->info.addr.pci.slot = pcislot; - net->info.addr.pci.bus = pcibus; + net->info.addr.pci.bus = bus; + net->info.addr.pci.slot = slot; net->info.addr.pci.function = function; if (!config) @@ -565,7 +565,7 @@ bhyveParseBhyvePCIArg(virDomainDefPtr def, char *slotdef = NULL; char *emulation = NULL; char *conf = NULL; - unsigned pcislot, bus, function; + unsigned bus, slot, function; separator = strchr(arg, ','); @@ -584,35 +584,35 @@ bhyveParseBhyvePCIArg(virDomainDefPtr def, emulation = g_strdup(separator); } - if (bhyveParsePCISlot(slotdef, &pcislot, &bus, &function) < 0) + if (bhyveParsePCISlot(slotdef, &bus, &slot, &function) < 0) goto error; if (STREQ(emulation, "ahci-cd")) - bhyveParsePCIDisk(def, caps, pcislot, bus, function, + bhyveParsePCIDisk(def, caps, bus, slot, function, VIR_DOMAIN_DISK_BUS_SATA, VIR_DOMAIN_DISK_DEVICE_CDROM, nvirtiodisk, nahcidisk, conf); else if (STREQ(emulation, "ahci-hd")) - bhyveParsePCIDisk(def, caps, pcislot, bus, function, + bhyveParsePCIDisk(def, caps, bus, slot, function, VIR_DOMAIN_DISK_BUS_SATA, VIR_DOMAIN_DISK_DEVICE_DISK, nvirtiodisk, nahcidisk, conf); else if (STREQ(emulation, "virtio-blk")) - bhyveParsePCIDisk(def, caps, pcislot, bus, function, + bhyveParsePCIDisk(def, caps, bus, slot, function, VIR_DOMAIN_DISK_BUS_VIRTIO, VIR_DOMAIN_DISK_DEVICE_DISK, nvirtiodisk, nahcidisk, conf); else if (STREQ(emulation, "virtio-net")) - bhyveParsePCINet(def, xmlopt, caps, pcislot, bus, function, + bhyveParsePCINet(def, xmlopt, caps, bus, slot, function, VIR_DOMAIN_NET_MODEL_VIRTIO, conf); else if (STREQ(emulation, "e1000")) - bhyveParsePCINet(def, xmlopt, caps, pcislot, bus, function, + bhyveParsePCINet(def, xmlopt, caps, bus, slot, function, VIR_DOMAIN_NET_MODEL_E1000, conf); VIR_FREE(emulation); -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:21AM -0500, Ryan Moeller wrote:
This *is* a no-op, but there was a period of sickening dread while auditing to be sure that no actual confusion between bus and slot had occurred. I hope to avoid that by following the conventional order.
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_parse_command.c | 34 ++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Handle PCI passthrough and virtio-scsi using hostdev devices. Example PCI passthrough: domain xml snippet ``` <memoryBacking> <locked/> </memoryBacking> <devices> <hostdev mode='subsystem' type='pci'> <source> <address domain='0x0000' bus='0x06' slot='0x02' function='0x00'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x00'/> </hostdev> </devices> ``` loader.conf snippet ``` vmm_load="YES" pptdevs="6/2/0" ``` Example SCSI passthrough: domain xml snippet ``` <hostdev mode='subsystem' type='scsi_ctl' model='virtio'> <source protocol='ioctl' pp='5' vp='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x00'/> </hostdev> ``` ctl.conf snippet ``` portal-group "pg0" { discovery-auth-group "no-authentication" listen "127.0.0.1" } target iqn.2020-01.com.example:target0 { auth-group "no-authentication" portal-group "pg0" port ioctl/5/0 lun 0 { path "/dev/zvol/storage/lun0" } lun 1 { path "/dev/zvol/storage/lun1" } lun 2 { path "/dev/zvol/storage/lun2" } lun 3 { path "/dev/zvol/storage/lun3" } } ``` Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- docs/schemas/domaincommon.rng | 30 ++++ src/bhyve/bhyve_capabilities.c | 14 ++ src/bhyve/bhyve_capabilities.h | 1 + src/bhyve/bhyve_command.c | 121 ++++++++++++++++ src/bhyve/bhyve_parse_command.c | 90 ++++++++++++ src/conf/domain_audit.c | 5 + src/conf/domain_conf.c | 131 ++++++++++++++++++ src/conf/domain_conf.h | 29 +++- src/conf/virconftypes.h | 3 + src/qemu/qemu_command.c | 2 + src/qemu/qemu_domain.c | 5 + src/qemu/qemu_hostdev.c | 1 + src/qemu/qemu_hotplug.c | 2 + src/qemu/qemu_migration.c | 1 + src/security/security_apparmor.c | 1 + src/security/security_dac.c | 28 ++++ src/security/security_selinux.c | 8 ++ .../bhyveargv2xml-passthru.args | 8 ++ .../bhyveargv2xml-passthru.xml | 26 ++++ .../bhyveargv2xml-virtio-scsi.args | 9 ++ .../bhyveargv2xml-virtio-scsi.xml | 20 +++ tests/bhyveargv2xmltest.c | 2 + .../bhyvexml2argv-passthru.args | 11 ++ .../bhyvexml2argv-passthru.ldargs | 1 + .../bhyvexml2argv-passthru.xml | 22 +++ .../bhyvexml2argv-virtio-scsi.args | 9 ++ .../bhyvexml2argv-virtio-scsi.ldargs | 1 + .../bhyvexml2argv-virtio-scsi.xml | 21 +++ tests/bhyvexml2argvtest.c | 4 +- .../bhyvexml2xmlout-passthru.xml | 29 ++++ .../bhyvexml2xmlout-virtio-scsi.xml | 23 +++ tests/bhyvexml2xmltest.c | 2 + 32 files changed, 658 insertions(+), 2 deletions(-) create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.args create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.xml create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.ldargs create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.args create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.ldargs create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.xml create mode 100644 tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml create mode 100644 tests/bhyvexml2xmloutdata/bhyvexml2xmlout-virtio-scsi.xml diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng index c00ace7d9c..e5250e1f32 100644 --- a/docs/schemas/domaincommon.rng +++ b/docs/schemas/domaincommon.rng @@ -4697,6 +4697,7 @@ </attribute> </optional> <choice> + <ref name="hostdevsubsysctl"/> <ref name="hostdevsubsyspci"/> <ref name="hostdevsubsysusb"/> <ref name="hostdevsubsysscsi"/> @@ -4723,6 +4724,34 @@ </define> + <define name="hostdevsubsysctl"> + <attribute name="type"> + <value>scsi_ctl</value> + </attribute> + <optional> + <attribute name="model"> + <choice> + <value>virtio</value> + </choice> + </attribute> + </optional> + <element name="source"> + <choice> + <group> + <attribute name="protocol"> + <value>ioctl</value> <!-- ioctl, required --> + </attribute> + <attribute name="pp"> + <data type="unsignedInt"/> + </attribute> + <attribute name="vp"> + <data type="unsignedInt"/> + </attribute> + </group> + </choice> + </element> + </define> + <define name="hostdevsubsyspci"> <attribute name="type"> <value>pci</value> @@ -4734,6 +4763,7 @@ <choice> <value>kvm</value> <value>vfio</value> + <value>vmm</value> <value>xen</value> </choice> </attribute> diff --git a/src/bhyve/bhyve_capabilities.c b/src/bhyve/bhyve_capabilities.c index fb8829d571..fb6be0aaba 100644 --- a/src/bhyve/bhyve_capabilities.c +++ b/src/bhyve/bhyve_capabilities.c @@ -323,6 +323,17 @@ bhyveProbeCapsXHCIController(unsigned int *caps, char *binary) } +static int +bhyveProbeCapsVirtioSCSI(unsigned int *caps, char *binary) +{ + return bhyveProbeCapsDeviceHelper(caps, binary, + "-s", + "0,virtio-scsi", + "pci slot 0:0: unknown device \"virtio-scsi\"", + BHYVE_CAP_VIRTIOSCSI); +} + + int virBhyveProbeCaps(unsigned int *caps) { @@ -351,6 +362,9 @@ virBhyveProbeCaps(unsigned int *caps) if ((ret = bhyveProbeCapsXHCIController(caps, binary))) goto out; + if ((ret = bhyveProbeCapsVirtioSCSI(caps, binary))) + goto out; + out: VIR_FREE(binary); return ret; diff --git a/src/bhyve/bhyve_capabilities.h b/src/bhyve/bhyve_capabilities.h index 12926cf423..bb62bdfb15 100644 --- a/src/bhyve/bhyve_capabilities.h +++ b/src/bhyve/bhyve_capabilities.h @@ -49,6 +49,7 @@ typedef enum { BHYVE_CAP_FBUF = 1 << 4, BHYVE_CAP_XHCI = 1 << 5, BHYVE_CAP_CPUTOPOLOGY = 1 << 6, + BHYVE_CAP_VIRTIOSCSI = 1 << 7, } virBhyveCapsFlags; int virBhyveProbeGrubCaps(virBhyveGrubCapsFlags *caps); diff --git a/src/bhyve/bhyve_command.c b/src/bhyve/bhyve_command.c index 5b1d80083a..9f4030f6a3 100644 --- a/src/bhyve/bhyve_command.c +++ b/src/bhyve/bhyve_command.c @@ -362,6 +362,123 @@ bhyveBuildControllerArgStr(const virDomainDef *def, return 0; } +static int +bhyveBuildHostdevSubsysPCIArgStr(const virDomainDef *def, + virDomainHostdevDefPtr dev, + bhyveConnPtr driver G_GNUC_UNUSED, + virCommandPtr cmd) +{ + virDomainHostdevSubsysPCIPtr pcisrc = &dev->source.subsys.u.pci; + + switch (pcisrc->backend) { + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_DEFAULT: + pcisrc->backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM; + G_GNUC_FALLTHROUGH; + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM: + if (!def->mem.locked) { + /* TODO: maybe just configure it automatically? */ + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("guest memory must be locked (wired) for " + "PCI passthrough")); + return -1; + } + virCommandAddArg(cmd, "-s"); + virCommandAddArgFormat(cmd, "%u:%u:%u,passthru,%u/%u/%u", + dev->info->addr.pci.bus, + dev->info->addr.pci.slot, + dev->info->addr.pci.function, + pcisrc->addr.bus, + pcisrc->addr.slot, + pcisrc->addr.function); + break; + default: + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported hostdev pci backend")); + return -1; + } + return 0; +} + +static int +bhyveBuildHostdevSubsysSCSICTLArgStr(const virDomainDef *def G_GNUC_UNUSED, + virDomainHostdevDefPtr dev, + bhyveConnPtr driver, + virCommandPtr cmd) +{ + virDomainHostdevSubsysSCSICTLPtr ctlsrc = &dev->source.subsys.u.scsi_ctl; + + /* Actually CAM Target Layer (CTL), not VHost on FreeBSD. */ + if (!(bhyveDriverGetBhyveCaps(driver) & BHYVE_CAP_VIRTIOSCSI)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("Installed bhyve binary does not support " + "defining virtio-scsi devices")); + return -1; + } + if (ctlsrc->protocol != + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_IOCTL) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported hostdev scsi_ctl protocol")); + return -1; + } + switch (ctlsrc->model) { + case VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_DEFAULT: + ctlsrc->model = + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_VIRTIO; + G_GNUC_FALLTHROUGH; + case VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_VIRTIO: + virCommandAddArg(cmd, "-s"); + virCommandAddArgFormat(cmd, "%u:%u,virtio-scsi,/dev/cam/ctl%u.%u", + dev->info->addr.pci.slot, + dev->info->addr.pci.function, + ctlsrc->pp, ctlsrc->vp); + break; + default: + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported hostdev scsi_ctl model")); + return -1; + } + return 0; +} + +static int +bhyveBuildHostdevSubsysArgStr(const virDomainDef *def, + virDomainHostdevDefPtr dev, + bhyveConnPtr driver, + virCommandPtr cmd) +{ + switch (dev->source.subsys.type) { + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI: + if (bhyveBuildHostdevSubsysPCIArgStr(def, dev, driver, cmd) < 0) + return -1; + break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + if (bhyveBuildHostdevSubsysSCSICTLArgStr(def, dev, driver, cmd) < 0) + return -1; + break; + default: + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported hostdev subsystem type")); + return -1; + } + return 0; +} + +static int +bhyveBuildHostdevArgStr(const virDomainDef *def, + virDomainHostdevDefPtr dev, + bhyveConnPtr driver, + virCommandPtr cmd) +{ + switch (dev->mode) { + case VIR_DOMAIN_HOSTDEV_MODE_SUBSYS: + return bhyveBuildHostdevSubsysArgStr(def, dev, driver, cmd); + default: + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unsupported hostdev device mode")); + return -1; + } +} + static int bhyveBuildLPCArgStr(const virDomainDef *def G_GNUC_UNUSED, virCommandPtr cmd) @@ -613,6 +730,10 @@ virBhyveProcessBuildBhyveCmd(bhyveConnPtr driver, virDomainDefPtr def, if (bhyveBuildDiskArgStr(def, def->disks[i], cmd) < 0) goto error; } + for (i = 0; i < def->nhostdevs; i++) { + if (bhyveBuildHostdevArgStr(def, def->hostdevs[i], driver, cmd) < 0) + goto error; + } if (def->ngraphics && def->nvideos) { if (def->ngraphics == 1 && def->nvideos == 1) { diff --git a/src/bhyve/bhyve_parse_command.c b/src/bhyve/bhyve_parse_command.c index 76423730d9..d69d23ace8 100644 --- a/src/bhyve/bhyve_parse_command.c +++ b/src/bhyve/bhyve_parse_command.c @@ -476,6 +476,92 @@ bhyveParsePCIDisk(virDomainDefPtr def, return -1; } +static int +bhyveParsePCIPassthru(virDomainDefPtr def, + unsigned vmbus, + unsigned vmslot, + unsigned vmfunction, + char *config) +{ + /* -s bus:slot:function,passthru,BUS/SLOT/FUNCTION */ + virDomainHostdevDefPtr dev = NULL; + virDomainHostdevSubsysPCIPtr pcisrc = NULL; + unsigned hostbus, hostslot, hostfunction; + + hostslot = hostbus = hostfunction = 0; + if (sscanf(config, "%u/%u/%u", &hostbus, &hostslot, &hostfunction) != 3) + return -1; + + if ((dev = virDomainHostdevDefNew()) == NULL) + return -1; + dev->info->type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; + dev->info->addr.pci.bus = vmbus; + dev->info->addr.pci.slot = vmslot; + dev->info->addr.pci.function = vmfunction; + dev->mode = VIR_DOMAIN_HOSTDEV_MODE_SUBSYS; + dev->source.subsys.type = VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI; + pcisrc = &dev->source.subsys.u.pci; + pcisrc->backend = VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM; + pcisrc->addr.bus = hostbus; + pcisrc->addr.slot = hostslot; + pcisrc->addr.function = hostfunction; + + if (VIR_APPEND_ELEMENT(def->hostdevs, def->nhostdevs, dev) < 0) + goto error; + + return 0; + + error: + virDomainHostdevDefFree(dev); + return -1; +} + +static int +bhyveParseSCSICTL(virDomainDefPtr def, + unsigned bus, + unsigned slot, + unsigned function, + char *config) +{ + /* -s slot,virtio-scsi,[dev=]/dev/cam/ctlPP.VP[,scsi-device-options] */ + virDomainHostdevDefPtr dev = NULL; + virDomainHostdevSubsysSCSICTLPtr ctlsrc = NULL; + unsigned pp, vp; + + /* Skip [dev=] if present. */ + if (STRPREFIX(config, "dev=")) + config = strchr(config, '=') + 1; + + pp = vp = 0; + if (sscanf(config, "/dev/cam/ctl%u.%u", &pp, &vp) != 2) + return -1; + + if ((dev = virDomainHostdevDefNew()) == NULL) + return -1; + dev->info->type = VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI; + dev->info->addr.pci.bus = bus; + dev->info->addr.pci.slot = slot; + dev->info->addr.pci.function = function; + dev->mode = VIR_DOMAIN_HOSTDEV_MODE_SUBSYS; + dev->source.subsys.type = VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL; + ctlsrc = &dev->source.subsys.u.scsi_ctl; + ctlsrc->model = + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_VIRTIO; + ctlsrc->protocol = + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_IOCTL; + ctlsrc->pp = pp; + ctlsrc->vp = vp; + + if (VIR_APPEND_ELEMENT(def->hostdevs, def->nhostdevs, dev) < 0) + goto error; + + return 0; + + error: + virDomainHostdevDefFree(dev); + return -1; +} + static int bhyveParsePCINet(virDomainDefPtr def, virDomainXMLOptionPtr xmlopt, @@ -601,6 +687,8 @@ bhyveParseBhyvePCIArg(virDomainDefPtr def, nvirtiodisk, nahcidisk, conf); + else if (STREQ(emulation, "passthru")) + bhyveParsePCIPassthru(def, bus, slot, function, conf); else if (STREQ(emulation, "virtio-blk")) bhyveParsePCIDisk(def, caps, bus, slot, function, VIR_DOMAIN_DISK_BUS_VIRTIO, @@ -611,6 +699,8 @@ bhyveParseBhyvePCIArg(virDomainDefPtr def, else if (STREQ(emulation, "virtio-net")) bhyveParsePCINet(def, xmlopt, caps, bus, slot, function, VIR_DOMAIN_NET_MODEL_VIRTIO, conf); + else if (STREQ(emulation, "virtio-scsi")) + bhyveParseSCSICTL(def, bus, slot, function, conf); else if (STREQ(emulation, "e1000")) bhyveParsePCINet(def, xmlopt, caps, bus, slot, function, VIR_DOMAIN_NET_MODEL_E1000, conf); diff --git a/src/conf/domain_audit.c b/src/conf/domain_audit.c index 1b0abb21a0..7322eca80c 100644 --- a/src/conf/domain_audit.c +++ b/src/conf/domain_audit.c @@ -348,6 +348,7 @@ virDomainAuditHostdev(virDomainObjPtr vm, virDomainHostdevDefPtr hostdev, virDomainHostdevSubsysUSBPtr usbsrc = &hostdev->source.subsys.u.usb; virDomainHostdevSubsysPCIPtr pcisrc = &hostdev->source.subsys.u.pci; virDomainHostdevSubsysSCSIPtr scsisrc = &hostdev->source.subsys.u.scsi; + virDomainHostdevSubsysSCSICTLPtr ctlsrc = &hostdev->source.subsys.u.scsi_ctl; virDomainHostdevSubsysSCSIVHostPtr hostsrc = &hostdev->source.subsys.u.scsi_host; virDomainHostdevSubsysMediatedDevPtr mdevsrc = &hostdev->source.subsys.u.mdev; @@ -387,6 +388,10 @@ virDomainAuditHostdev(virDomainObjPtr vm, virDomainHostdevDefPtr hostdev, } break; } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + address = g_strdup_printf("/dev/cam/ctl%u.%u", + ctlsrc->pp, ctlsrc->vp); + break; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: address = g_strdup(hostsrc->wwpn); break; diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index cef49df3f8..12f8bb43c0 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -898,6 +898,7 @@ VIR_ENUM_IMPL(virDomainHostdevSubsys, "usb", "pci", "scsi", + "scsi_ctl", "scsi_host", "mdev", ); @@ -908,6 +909,7 @@ VIR_ENUM_IMPL(virDomainHostdevSubsysPCIBackend, "kvm", "vfio", "xen", + "vmm", ); VIR_ENUM_IMPL(virDomainHostdevSubsysSCSIProtocol, @@ -916,6 +918,18 @@ VIR_ENUM_IMPL(virDomainHostdevSubsysSCSIProtocol, "iscsi", ); +VIR_ENUM_IMPL(virDomainHostdevSubsysSCSICTLProtocol, + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_LAST, + "none", + "ioctl", +); + +VIR_ENUM_IMPL(virDomainHostdevSubsysSCSICTLModel, + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_LAST, + "default", + "virtio", +); + VIR_ENUM_IMPL(virDomainHostdevSubsysSCSIHostProtocol, VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_HOST_PROTOCOL_TYPE_LAST, "none", @@ -2958,6 +2972,7 @@ void virDomainHostdevDefClear(virDomainHostdevDefPtr def) break; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_USB: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI: + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: break; @@ -5046,6 +5061,7 @@ virDomainHostdevDefPostParse(virDomainHostdevDefPtr dev, case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_USB: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI: + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: break; @@ -6484,6 +6500,16 @@ virDomainHostdevDefValidate(const virDomainHostdevDef *hostdev) return -1; } break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + if (hostdev->info->type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE && + hostdev->info->type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_UNASSIGNED && + hostdev->info->type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("SCSI CTL host devices must use 'pci' or " + "'unassigned' address type")); + return -1; + } + break; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: if (hostdev->info->type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE && hostdev->info->type != VIR_DOMAIN_DEVICE_ADDRESS_TYPE_PCI && @@ -8320,6 +8346,64 @@ virDomainHostdevSubsysSCSIVHostDefParseXML(xmlNodePtr sourcenode, return 0; } +static int +virDomainHostdevSubsysSCSICTLDefParseXML(xmlNodePtr sourcenode, + virDomainHostdevDefPtr def) +{ + virDomainHostdevSubsysSCSICTLPtr ctlsrc = &def->source.subsys.u.scsi_ctl; + g_autofree char *protocol = NULL; + g_autofree char *pp = NULL; + g_autofree char *vp = NULL; + + if (!(protocol = virXMLPropString(sourcenode, "protocol"))) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("Missing scsi_ctl subsystem protocol")); + return -1; + } + + if ((ctlsrc->protocol = + virDomainHostdevSubsysSCSICTLProtocolTypeFromString(protocol)) <= 0) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Unknown scsi_ctl subsystem protocol '%s'"), + protocol); + return -1; + } + + switch ((virDomainHostdevSubsysSCSICTLProtocolType) ctlsrc->protocol) { + case VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_IOCTL: + if (!(pp = virXMLPropString(sourcenode, "pp"))) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("missing scsi_ctl hostdev source pp")); + return -1; + } + if (virStrToLong_ui(pp, NULL, 10, &ctlsrc->pp) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Cannot parse scsi_ctl hostdev source 'pp' attribute")); + return -1; + } + if (!(vp = virXMLPropString(sourcenode, "vp"))) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("missing scsi_ctl hostdev source vp")); + return -1; + } + if (virStrToLong_ui(vp, NULL, 10, &ctlsrc->vp) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Cannot parse scsi_ctl hostdev source 'vp' attribute")); + return -1; + } + break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_NONE: + case VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_LAST: + virReportError(VIR_ERR_XML_ERROR, + _("Invalid hostdev protocol '%s'"), + virDomainHostdevSubsysSCSICTLProtocolTypeToString(ctlsrc->protocol)); + return -1; + break; + } + + return 0; +} + static int virDomainHostdevSubsysMediatedDevDefParseXML(virDomainHostdevDefPtr def, xmlXPathContextPtr ctxt) @@ -8363,6 +8447,7 @@ virDomainHostdevDefParseXMLSubsys(xmlNodePtr node, int backend; virDomainHostdevSubsysPCIPtr pcisrc = &def->source.subsys.u.pci; virDomainHostdevSubsysSCSIPtr scsisrc = &def->source.subsys.u.scsi; + virDomainHostdevSubsysSCSICTLPtr scsictlsrc = &def->source.subsys.u.scsi_ctl; virDomainHostdevSubsysSCSIVHostPtr scsihostsrc = &def->source.subsys.u.scsi_host; virDomainHostdevSubsysMediatedDevPtr mdevsrc = &def->source.subsys.u.mdev; g_autofree char *managed = NULL; @@ -8453,6 +8538,7 @@ virDomainHostdevDefParseXMLSubsys(xmlNodePtr node, } if (def->source.subsys.type != VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV && + def->source.subsys.type != VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL && def->source.subsys.type != VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST) { if (model) { virReportError(VIR_ERR_XML_ERROR, @@ -8471,6 +8557,14 @@ virDomainHostdevDefParseXMLSubsys(xmlNodePtr node, model); return -1; } + } else if (def->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL) { + if (model && + ((scsictlsrc->model = virDomainHostdevSubsysSCSICTLModelTypeFromString(model)) < 0)) { + virReportError(VIR_ERR_XML_ERROR, + _("unknown hostdev model '%s'"), + model); + return -1; + } } else if (def->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV) { if (!model) { virReportError(VIR_ERR_XML_ERROR, "%s", @@ -8533,10 +8627,16 @@ virDomainHostdevDefParseXMLSubsys(xmlNodePtr node, return -1; break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + if (virDomainHostdevSubsysSCSICTLDefParseXML(sourcenode, def) < 0) + return -1; + break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: if (virDomainHostdevSubsysSCSIVHostDefParseXML(sourcenode, def) < 0) return -1; break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: if (virDomainHostdevSubsysMediatedDevDefParseXML(def, ctxt) < 0) return -1; @@ -15914,6 +16014,7 @@ virDomainHostdevDefParseXML(virDomainXMLOptionPtr xmlopt, case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_USB: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI: + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: @@ -16963,6 +17064,13 @@ virDomainHostdevMatchSubsys(virDomainHostdevDefPtr a, return virDomainHostdevMatchSubsysSCSIiSCSI(a, b); else return virDomainHostdevMatchSubsysSCSIHost(a, b); + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + return ((a->source.subsys.u.scsi_ctl.protocol == + b->source.subsys.u.scsi_ctl.protocol) && + (a->source.subsys.u.scsi_ctl.pp == + b->source.subsys.u.scsi_ctl.pp) && + (a->source.subsys.u.scsi_ctl.vp == + b->source.subsys.u.scsi_ctl.vp)) ? 1 : 0; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: if (a->source.subsys.u.scsi_host.protocol != b->source.subsys.u.scsi_host.protocol) @@ -25354,6 +25462,7 @@ virDomainHostdevDefFormatSubsys(virBufferPtr buf, virDomainHostdevSubsysUSBPtr usbsrc = &def->source.subsys.u.usb; virDomainHostdevSubsysPCIPtr pcisrc = &def->source.subsys.u.pci; virDomainHostdevSubsysSCSIPtr scsisrc = &def->source.subsys.u.scsi; + virDomainHostdevSubsysSCSICTLPtr ctlsrc = &def->source.subsys.u.scsi_ctl; virDomainHostdevSubsysSCSIVHostPtr hostsrc = &def->source.subsys.u.scsi_host; virDomainHostdevSubsysMediatedDevPtr mdevsrc = &def->source.subsys.u.mdev; virDomainHostdevSubsysSCSIHostPtr scsihostsrc = &scsisrc->u.host; @@ -25396,6 +25505,15 @@ virDomainHostdevDefFormatSubsys(virBufferPtr buf, protocol, iscsisrc->src->path); } + if (def->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL) { + const char *protocol = + virDomainHostdevSubsysSCSICTLProtocolTypeToString(ctlsrc->protocol); + closedSource = true; + + virBufferAsprintf(buf, " protocol='%s' pp='%u' vp='%u'/", + protocol, ctlsrc->pp, ctlsrc->vp); + } + if (def->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST) { const char *protocol = virDomainHostdevSubsysSCSIHostProtocolTypeToString(hostsrc->protocol); @@ -25457,6 +25575,7 @@ virDomainHostdevDefFormatSubsys(virBufferPtr buf, scsihostsrc->unit); } break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: break; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: @@ -27560,6 +27679,7 @@ virDomainHostdevDefFormat(virBufferPtr buf, virDomainHostdevSubsysSCSIPtr scsisrc = &def->source.subsys.u.scsi; virDomainHostdevSubsysMediatedDevPtr mdevsrc = &def->source.subsys.u.mdev; virDomainHostdevSubsysSCSIVHostPtr scsihostsrc = &def->source.subsys.u.scsi_host; + virDomainHostdevSubsysSCSICTLPtr scsictlsrc = &def->source.subsys.u.scsi_ctl; const char *type; if (!mode) { @@ -27610,6 +27730,12 @@ virDomainHostdevDefFormat(virBufferPtr buf, virTristateBoolTypeToString(scsisrc->rawio)); } + if (def->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL && + scsictlsrc->model) { + virBufferAsprintf(buf, " model='%s'", + virDomainHostdevSubsysSCSICTLModelTypeToString(scsictlsrc->model)); + } + if (def->source.subsys.type == VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST && scsihostsrc->model) { virBufferAsprintf(buf, " model='%s'", @@ -31142,6 +31268,11 @@ virDomainNetDefActualToNetworkPort(virDomainDefPtr dom, port->plug.hostdevpci.driver = VIR_NETWORK_FORWARD_DRIVER_NAME_VFIO; break; + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM: + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("Unexpected PCI backend 'vmm'")); + break; + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_XEN: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Unexpected PCI backend 'xen'")); diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index cdc4d25700..4e0fc01c2b 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -195,6 +195,7 @@ typedef enum { VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_USB, VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI, VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI, + VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL, VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST, VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV, @@ -203,10 +204,11 @@ typedef enum { /* the backend driver used for PCI hostdev devices */ typedef enum { - VIR_DOMAIN_HOSTDEV_PCI_BACKEND_DEFAULT, /* detect automatically, prefer VFIO */ + VIR_DOMAIN_HOSTDEV_PCI_BACKEND_DEFAULT, /* detect automatically, prefer VFIO or VMM */ VIR_DOMAIN_HOSTDEV_PCI_BACKEND_KVM, /* force legacy kvm style */ VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VFIO, /* force vfio */ VIR_DOMAIN_HOSTDEV_PCI_BACKEND_XEN, /* force legacy xen style, use pciback */ + VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM, /* force vmm (FreeBSD bhyve) */ VIR_DOMAIN_HOSTDEV_PCI_BACKEND_TYPE_LAST } virDomainHostdevSubsysPCIBackendType; @@ -258,6 +260,30 @@ struct _virDomainHostdevSubsysSCSI { } u; }; +typedef enum { + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_NONE, + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_IOCTL, + + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_PROTOCOL_TYPE_LAST, +} virDomainHostdevSubsysSCSICTLProtocolType; + +VIR_ENUM_DECL(virDomainHostdevSubsysSCSICTLProtocol); + +typedef enum { + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_DEFAULT, + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_VIRTIO, + + VIR_DOMAIN_HOSTDEV_SUBSYS_SCSI_CTL_MODEL_TYPE_LAST, +} virDomainHostdevSubsysSCSICTLModelType; + +VIR_ENUM_DECL(virDomainHostdevSubsysSCSICTLModel); + +struct _virDomainHostdevSubsysSCSICTL { + int protocol; /* enum virDomainHostdevSubsysSCSICTLProtocolType */ + unsigned pp, vp; + int model; /* enum virDomainHostdevSubsysSCSICTLModelType */ +}; + struct _virDomainHostdevSubsysMediatedDev { int model; /* enum virMediatedDeviceModelType */ int display; /* virTristateSwitch */ @@ -298,6 +324,7 @@ struct _virDomainHostdevSubsys { virDomainHostdevSubsysPCI pci; virDomainHostdevSubsysSCSI scsi; virDomainHostdevSubsysSCSIVHost scsi_host; + virDomainHostdevSubsysSCSICTL scsi_ctl; virDomainHostdevSubsysMediatedDev mdev; } u; }; diff --git a/src/conf/virconftypes.h b/src/conf/virconftypes.h index 1c62cde251..66cf40f8f0 100644 --- a/src/conf/virconftypes.h +++ b/src/conf/virconftypes.h @@ -180,6 +180,9 @@ typedef virDomainHostdevSubsysPCI *virDomainHostdevSubsysPCIPtr; typedef struct _virDomainHostdevSubsysSCSI virDomainHostdevSubsysSCSI; typedef virDomainHostdevSubsysSCSI *virDomainHostdevSubsysSCSIPtr; +typedef struct _virDomainHostdevSubsysSCSICTL virDomainHostdevSubsysSCSICTL; +typedef virDomainHostdevSubsysSCSICTL *virDomainHostdevSubsysSCSICTLPtr; + typedef struct _virDomainHostdevSubsysSCSIHost virDomainHostdevSubsysSCSIHost; typedef virDomainHostdevSubsysSCSIHost *virDomainHostdevSubsysSCSIHostPtr; diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index f69a9e651c..fdd59a35a4 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -4677,6 +4677,7 @@ qemuBuildPCIHostdevDevStr(const virDomainDef *def, case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_KVM: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_DEFAULT: + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_XEN: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_TYPE_LAST: virReportError(VIR_ERR_INTERNAL_ERROR, @@ -5439,6 +5440,7 @@ qemuBuildHostdevCommandLine(virCommandPtr cmd, break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: break; } diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index af6817cc05..3ebb864b4c 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -6674,8 +6674,11 @@ qemuDomainDeviceDefValidateHostdev(const virDomainHostdevDef *hostdev, return -1; } break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: return qemuDomainMdevDefValidate(hostdev, def, qemuCaps); + + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: default: virReportEnumRangeError(virDomainHostdevSubsysType, @@ -14031,6 +14034,8 @@ qemuDomainGetHostdevPath(virDomainHostdevDefPtr dev, perm = VIR_CGROUP_DEVICE_RW; break; + + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: break; } diff --git a/src/qemu/qemu_hostdev.c b/src/qemu/qemu_hostdev.c index 1774850640..0e1432f23d 100644 --- a/src/qemu/qemu_hostdev.c +++ b/src/qemu/qemu_hostdev.c @@ -203,6 +203,7 @@ qemuHostdevPreparePCIDevicesCheckSupport(virDomainHostdevDefPtr *hostdevs, return false; break; + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_XEN: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_TYPE_LAST: break; diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 9800491755..94fa64f216 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -1576,6 +1576,7 @@ qemuDomainAttachHostPCIDevice(virQEMUDriverPtr driver, case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_KVM: break; + case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VMM: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_XEN: case VIR_DOMAIN_HOSTDEV_PCI_BACKEND_TYPE_LAST: virReportError(VIR_ERR_CONFIG_UNSUPPORTED, @@ -4535,6 +4536,7 @@ qemuDomainRemoveHostDevice(virQEMUDriverPtr driver, case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: qemuDomainRemoveMediatedDevice(driver, vm, hostdev); break; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: break; } diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a307c5ebe2..93144b741f 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1105,6 +1105,7 @@ qemuMigrationSrcIsAllowedHostdev(const virDomainDef *def) continue; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI: + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_MDEV: virReportError(VIR_ERR_OPERATION_UNSUPPORTED, diff --git a/src/security/security_apparmor.c b/src/security/security_apparmor.c index ca02631f7f..eadf2580b6 100644 --- a/src/security/security_apparmor.c +++ b/src/security/security_apparmor.c @@ -959,6 +959,7 @@ AppArmorSetSecurityHostdevLabel(virSecurityManagerPtr mgr, break; } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_LAST: ret = 0; break; diff --git a/src/security/security_dac.c b/src/security/security_dac.c index d75b18170b..de85e53a9a 100644 --- a/src/security/security_dac.c +++ b/src/security/security_dac.c @@ -1212,6 +1212,7 @@ virSecurityDACSetHostdevLabel(virSecurityManagerPtr mgr, virDomainHostdevSubsysUSBPtr usbsrc = &dev->source.subsys.u.usb; virDomainHostdevSubsysPCIPtr pcisrc = &dev->source.subsys.u.pci; virDomainHostdevSubsysSCSIPtr scsisrc = &dev->source.subsys.u.scsi; + virDomainHostdevSubsysSCSICTLPtr ctlsrc = &dev->source.subsys.u.scsi_ctl; virDomainHostdevSubsysSCSIVHostPtr hostsrc = &dev->source.subsys.u.scsi_host; virDomainHostdevSubsysMediatedDevPtr mdevsrc = &dev->source.subsys.u.mdev; int ret = -1; @@ -1300,6 +1301,19 @@ virSecurityDACSetHostdevLabel(virSecurityManagerPtr mgr, break; } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: { + char *ctldev = g_strdup_printf("/dev/cam/ctl%u.%u", + ctlsrc->pp, ctlsrc->vp); + + if (!ctldev) + return -1; + + ret = virSecurityDACSetHostdevLabelHelper(ctldev, true, &cbdata); + + VIR_FREE(ctldev); + break; + } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: { virSCSIVHostDevicePtr host = virSCSIVHostDeviceNew(hostsrc->wwpn); @@ -1386,6 +1400,7 @@ virSecurityDACRestoreHostdevLabel(virSecurityManagerPtr mgr, virDomainHostdevSubsysUSBPtr usbsrc = &dev->source.subsys.u.usb; virDomainHostdevSubsysPCIPtr pcisrc = &dev->source.subsys.u.pci; virDomainHostdevSubsysSCSIPtr scsisrc = &dev->source.subsys.u.scsi; + virDomainHostdevSubsysSCSICTLPtr ctlsrc = &dev->source.subsys.u.scsi_ctl; virDomainHostdevSubsysSCSIVHostPtr hostsrc = &dev->source.subsys.u.scsi_host; virDomainHostdevSubsysMediatedDevPtr mdevsrc = &dev->source.subsys.u.mdev; int ret = -1; @@ -1463,6 +1478,19 @@ virSecurityDACRestoreHostdevLabel(virSecurityManagerPtr mgr, break; } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: { + char *ctldev = g_strdup_printf("/dev/cam/ctl%u.%u", + ctlsrc->pp, ctlsrc->vp); + + if (!ctldev) + return -1; + + ret = virSecurityDACRestoreFileLabel(mgr, ctldev); + + VIR_FREE(ctldev); + break; + } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: { virSCSIVHostDevicePtr host = virSCSIVHostDeviceNew(hostsrc->wwpn); diff --git a/src/security/security_selinux.c b/src/security/security_selinux.c index 3f6968a57a..96f483ee14 100644 --- a/src/security/security_selinux.c +++ b/src/security/security_selinux.c @@ -2149,6 +2149,10 @@ virSecuritySELinuxSetHostdevSubsysLabel(virSecurityManagerPtr mgr, break; } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + /* FreeBSD only */ + return -1; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: { virSCSIVHostDevicePtr host = virSCSIVHostDeviceNew(hostsrc->wwpn); @@ -2384,6 +2388,10 @@ virSecuritySELinuxRestoreHostdevSubsysLabel(virSecurityManagerPtr mgr, break; } + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_CTL: + /* FreeBSD only */ + return -1; + case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_SCSI_HOST: { virSCSIVHostDevicePtr host = virSCSIVHostDeviceNew(hostsrc->wwpn); diff --git a/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args new file mode 100644 index 0000000000..697bafd642 --- /dev/null +++ b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args @@ -0,0 +1,8 @@ +/usr/sbin/bhyve \ +-c 1 \ +-m 214 \ +-H \ +-P \ +-S \ +-s 0:0,hostbridge \ +-s 0:1:0,passthru,5/0/7 bhyve diff --git a/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml new file mode 100644 index 0000000000..af99279448 --- /dev/null +++ b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml @@ -0,0 +1,26 @@ +<domain type='bhyve'> + <name>bhyve</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <memoryBacking> + <locked/> + </memoryBacking> + <vcpu placement='static'>1</vcpu> + <os> + <type>hvm</type> + </os> + <clock offset='localtime'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>destroy</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <hostdev mode='subsystem' type='pci' managed='no'> + <driver name='vmm'/> + <source> + <address domain='0x0000' bus='0x05' slot='0x00' function='0x7'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> + </hostdev> + </devices> +</domain> diff --git a/tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.args b/tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.args new file mode 100644 index 0000000000..ae38208853 --- /dev/null +++ b/tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.args @@ -0,0 +1,9 @@ +/usr/sbin/bhyve \ +-c 1 \ +-m 214 \ +-H \ +-P \ +-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \ +-s 0:0,hostbridge \ +-s 1:0,lpc \ +-s 2:0,virtio-scsi,/dev/cam/ctl5.0 bhyve diff --git a/tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.xml b/tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.xml new file mode 100644 index 0000000000..23ad90a2a5 --- /dev/null +++ b/tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.xml @@ -0,0 +1,20 @@ +<domain type='bhyve'> + <name>bhyve</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type>hvm</type> + </os> + <clock offset='localtime'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>destroy</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <hostdev mode='subsystem' type='scsi_ctl' managed='no' model='virtio'> + <source protocol='ioctl' pp='5' vp='0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> + </hostdev> + </devices> +</domain> diff --git a/tests/bhyveargv2xmltest.c b/tests/bhyveargv2xmltest.c index 735cc4b338..1cfe4e3ddb 100644 --- a/tests/bhyveargv2xmltest.c +++ b/tests/bhyveargv2xmltest.c @@ -173,6 +173,8 @@ mymain(void) DO_TEST("ahci-hd"); DO_TEST("virtio-blk"); DO_TEST("virtio-net"); + DO_TEST("virtio-scsi"); + DO_TEST("passthru"); DO_TEST("e1000"); DO_TEST_WARN("virtio-net2"); DO_TEST_WARN("virtio-net3"); diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args new file mode 100644 index 0000000000..c268da957c --- /dev/null +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args @@ -0,0 +1,11 @@ +/usr/sbin/bhyve \ +-c 1 \ +-m 214 \ +-S \ +-u \ +-H \ +-P \ +-s 0:0,hostbridge \ +-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \ +-s 0:3:0,passthru,5/0/7 \ +-s 1,lpc bhyve diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.ldargs b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.ldargs new file mode 100644 index 0000000000..2995a4d0e7 --- /dev/null +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.ldargs @@ -0,0 +1 @@ +dummy \ No newline at end of file diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml new file mode 100644 index 0000000000..ba0744f35d --- /dev/null +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml @@ -0,0 +1,22 @@ +<domain type='bhyve'> + <name>bhyve</name> + <uuid>df3be7e7-a104-11e3-aeb0-50e5492bd3dc</uuid> + <memory>219136</memory> + <memoryBacking> + <locked/> + </memoryBacking> + <vcpu>1</vcpu> + <os> + <type>hvm</type> + <loader readonly='yes' type='pflash'>/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd</loader> + </os> + <devices> + <hostdev mode='subsystem' type='pci' managed='no'> + <driver name='vmm'/> + <source> + <address domain='0x0000' bus='0x05' slot='0x00' function='0x7'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </hostdev> + </devices> +</domain> diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.args b/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.args new file mode 100644 index 0000000000..90b82cc02b --- /dev/null +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.args @@ -0,0 +1,9 @@ +/usr/sbin/bhyve \ +-c 1 \ +-m 214 \ +-H \ +-P \ +-s 0:0,hostbridge \ +-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \ +-s 2:0,virtio-scsi,/dev/cam/ctl5.0 \ +-s 1,lpc bhyve diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.ldargs b/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.ldargs new file mode 100644 index 0000000000..421376db9e --- /dev/null +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.ldargs @@ -0,0 +1 @@ +dummy diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.xml b/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.xml new file mode 100644 index 0000000000..394ff20ffc --- /dev/null +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.xml @@ -0,0 +1,21 @@ +<domain type='bhyve'> + <name>bhyve</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type>hvm</type> + <loader readonly='yes' type='pflash'>/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd</loader> + </os> + <clock offset='localtime'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>destroy</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <hostdev mode='subsystem' type='scsi_ctl' model='virtio'> + <source protocol='ioctl' pp='5' vp='0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> + </hostdev> + </devices> +</domain> diff --git a/tests/bhyvexml2argvtest.c b/tests/bhyvexml2argvtest.c index 9e7eb218b8..7ce2d5c6b1 100644 --- a/tests/bhyvexml2argvtest.c +++ b/tests/bhyvexml2argvtest.c @@ -175,7 +175,7 @@ mymain(void) driver.bhyvecaps = BHYVE_CAP_RTC_UTC | BHYVE_CAP_AHCI32SLOT | \ BHYVE_CAP_NET_E1000 | BHYVE_CAP_LPC_BOOTROM | \ BHYVE_CAP_FBUF | BHYVE_CAP_XHCI | \ - BHYVE_CAP_CPUTOPOLOGY; + BHYVE_CAP_CPUTOPOLOGY | BHYVE_CAP_VIRTIOSCSI; DO_TEST("base"); DO_TEST("wired"); @@ -210,6 +210,8 @@ mymain(void) DO_TEST_FAILURE("cputopology-nvcpu-mismatch"); DO_TEST("commandline"); DO_TEST("msrs"); + DO_TEST("virtio-scsi"); + DO_TEST("passthru"); /* Address allocation tests */ DO_TEST("addr-single-sata-disk"); diff --git a/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml b/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml new file mode 100644 index 0000000000..0313fa0dfa --- /dev/null +++ b/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml @@ -0,0 +1,29 @@ +<domain type='bhyve'> + <name>bhyve</name> + <uuid>df3be7e7-a104-11e3-aeb0-50e5492bd3dc</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <memoryBacking> + <locked/> + </memoryBacking> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='x86_64'>hvm</type> + <loader readonly='yes' type='pflash'>/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd</loader> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <controller type='pci' index='0' model='pci-root'/> + <hostdev mode='subsystem' type='pci' managed='no'> + <driver name='vmm'/> + <source> + <address domain='0x0000' bus='0x05' slot='0x00' function='0x7'/> + </source> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </hostdev> + </devices> +</domain> diff --git a/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-virtio-scsi.xml b/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-virtio-scsi.xml new file mode 100644 index 0000000000..771591689a --- /dev/null +++ b/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-virtio-scsi.xml @@ -0,0 +1,23 @@ +<domain type='bhyve'> + <name>bhyve</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='x86_64'>hvm</type> + <loader readonly='yes' type='pflash'>/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd</loader> + <boot dev='hd'/> + </os> + <clock offset='localtime'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>destroy</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <controller type='pci' index='0' model='pci-root'/> + <hostdev mode='subsystem' type='scsi_ctl' managed='no' model='virtio'> + <source protocol='ioctl' pp='5' vp='0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> + </hostdev> + </devices> +</domain> diff --git a/tests/bhyvexml2xmltest.c b/tests/bhyvexml2xmltest.c index a0c20a14c1..3a2a46014a 100644 --- a/tests/bhyvexml2xmltest.c +++ b/tests/bhyvexml2xmltest.c @@ -83,6 +83,7 @@ mymain(void) DO_TEST_DIFFERENT("acpiapic"); DO_TEST_DIFFERENT("base"); DO_TEST_DIFFERENT("wired"); + DO_TEST_DIFFERENT("passthru"); DO_TEST_DIFFERENT("bhyveload-bootorder"); DO_TEST_DIFFERENT("bhyveload-bootorder1"); DO_TEST_DIFFERENT("bhyveload-bootorder2"); @@ -94,6 +95,7 @@ mymain(void) DO_TEST_DIFFERENT("disk-cdrom"); DO_TEST_DIFFERENT("disk-cdrom-grub"); DO_TEST_DIFFERENT("disk-virtio"); + DO_TEST_DIFFERENT("virtio-scsi"); DO_TEST_DIFFERENT("grub-bootorder"); DO_TEST_DIFFERENT("grub-bootorder2"); DO_TEST_DIFFERENT("grub-defaults"); -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:22AM -0500, Ryan Moeller wrote:
Handle PCI passthrough and virtio-scsi using hostdev devices.
Example PCI passthrough: domain xml snippet ``` <memoryBacking> <locked/> </memoryBacking> <devices> <hostdev mode='subsystem' type='pci'> <source> <address domain='0x0000' bus='0x06' slot='0x02' function='0x00'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x00'/> </hostdev> </devices> ``` loader.conf snippet ``` vmm_load="YES" pptdevs="6/2/0" ```
Example SCSI passthrough: domain xml snippet ``` <hostdev mode='subsystem' type='scsi_ctl' model='virtio'> <source protocol='ioctl' pp='5' vp='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x00'/> </hostdev> ``` ctl.conf snippet ``` portal-group "pg0" { discovery-auth-group "no-authentication" listen "127.0.0.1" }
target iqn.2020-01.com.example:target0 { auth-group "no-authentication" portal-group "pg0" port ioctl/5/0
lun 0 { path "/dev/zvol/storage/lun0" } lun 1 { path "/dev/zvol/storage/lun1" } lun 2 { path "/dev/zvol/storage/lun2" } lun 3 { path "/dev/zvol/storage/lun3" } } ```
Can you split this patch up into a few pieces. We generally want the changes to the XML parser/formatter to be separate from any driver code, as the first part. Then I'd suggest a separate patch for the PCI and the SCSI hostdev support in bhyve.
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- docs/schemas/domaincommon.rng | 30 ++++ src/bhyve/bhyve_capabilities.c | 14 ++ src/bhyve/bhyve_capabilities.h | 1 + src/bhyve/bhyve_command.c | 121 ++++++++++++++++ src/bhyve/bhyve_parse_command.c | 90 ++++++++++++ src/conf/domain_audit.c | 5 + src/conf/domain_conf.c | 131 ++++++++++++++++++ src/conf/domain_conf.h | 29 +++-
For the conf stuff we'll need a docs update in docs/formatdomain.html.in at the same time as this parser additions. If you can provide a little detail in the commit message on why the current SCSI hostdev stuff doesn't work for FreeBSD that'd be useful too.
src/conf/virconftypes.h | 3 + src/qemu/qemu_command.c | 2 + src/qemu/qemu_domain.c | 5 + src/qemu/qemu_hostdev.c | 1 + src/qemu/qemu_hotplug.c | 2 + src/qemu/qemu_migration.c | 1 + src/security/security_apparmor.c | 1 + src/security/security_dac.c | 28 ++++ src/security/security_selinux.c | 8 ++ .../bhyveargv2xml-passthru.args | 8 ++ .../bhyveargv2xml-passthru.xml | 26 ++++ .../bhyveargv2xml-virtio-scsi.args | 9 ++ .../bhyveargv2xml-virtio-scsi.xml | 20 +++ tests/bhyveargv2xmltest.c | 2 + [> .../bhyvexml2argv-passthru.args | 11 ++ .../bhyvexml2argv-passthru.ldargs | 1 + .../bhyvexml2argv-passthru.xml | 22 +++ .../bhyvexml2argv-virtio-scsi.args | 9 ++ .../bhyvexml2argv-virtio-scsi.ldargs | 1 + .../bhyvexml2argv-virtio-scsi.xml | 21 +++ tests/bhyvexml2argvtest.c | 4 +- .../bhyvexml2xmlout-passthru.xml | 29 ++++ .../bhyvexml2xmlout-virtio-scsi.xml | 23 +++ tests/bhyvexml2xmltest.c | 2 + 32 files changed, 658 insertions(+), 2 deletions(-) create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.args create mode 100644 tests/bhyveargv2xmldata/bhyveargv2xml-virtio-scsi.xml create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.ldargs create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.args create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.ldargs create mode 100644 tests/bhyvexml2argvdata/bhyvexml2argv-virtio-scsi.xml create mode 100644 tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml create mode 100644 tests/bhyvexml2xmloutdata/bhyvexml2xmlout-virtio-scsi.xml
I didn't see anything in the code is keeping track of in-use PCI devices. Does something else in FreeBSD guarantee that you won't have bad stuff happening if the PCI device is attempted to be assigned to 2 guests ? Also is it required to manually detach the host OS driver first, or is that automatic ? This ties into which 'managed=no|yes' attribute choices you should permit for the hostdev. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Can you split this patch up into a few pieces.
We generally want the changes to the XML parser/formatter to be separate from any driver code, as the first part.
Then I'd suggest a separate patch for the PCI and the SCSI hostdev support in bhyve.
For the conf stuff we'll need a docs update in docs/formatdomain.html.in at the same time as this parser additions.
If you can provide a little detail in the commit message on why the current SCSI hostdev stuff doesn't work for FreeBSD that'd be useful too.
Can do.
I didn't see anything in the code is keeping track of in-use PCI devices. Does something else in FreeBSD guarantee that you won't have bad stuff happening if the PCI device is attempted to be assigned to 2 guests ?
Also is it required to manually detach the host OS driver first, or is that automatic ? This ties into which 'managed=no|yes' attribute choices you should permit for the hostdev.
There is not automatic management for passing through devices. It is configured by a list of PCI devices that are to be reserved for passthrough at boot. The vmm module will attach the listed devices to the ppt driver before something else can try to take them. If the same device is assigned to multiple VMs then the bhyve command will error with an appropriate message when trying to start VMs after the first one takes the device. This seems fine to me. I have several VMs configured to use the same devices but only can run one at a time. This is for testing drivers in different stable branches, on different operating systems, etc. I could imagine some future extension of this work to enable libvirt to automatically configure devices for passthrough at next boot perhaps, but that is more of a convenience than a necessity. Right now passthrough can only be configured by manually putting the bhyve command line args in the config, so this change is already a definite improvement in terms of leveraging libvirt's facilities. -- Ryan Moeller iXsystems, Inc. OS Developer Email: ryan@iXsystems.com

It is possible to boot from virtio-scsi with UEFI-devel, for example. Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_command.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/src/bhyve/bhyve_command.c b/src/bhyve/bhyve_command.c index 9f4030f6a3..048c26733a 100644 --- a/src/bhyve/bhyve_command.c +++ b/src/bhyve/bhyve_command.c @@ -993,11 +993,8 @@ virBhyveGetBootDisk(virDomainDefPtr def) virDomainDiskDefPtr match = NULL; int boot_dev = -1; - if (def->ndisks < 1) { - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", - _("Domain should have at least one disk defined")); + if (def->ndisks == 0) return NULL; - } if (def->os.nBootDevs > 1) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:23AM -0500, Ryan Moeller wrote:
It is possible to boot from virtio-scsi with UEFI-devel, for example.
Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- src/bhyve/bhyve_command.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/src/bhyve/bhyve_command.c b/src/bhyve/bhyve_command.c index 9f4030f6a3..048c26733a 100644 --- a/src/bhyve/bhyve_command.c +++ b/src/bhyve/bhyve_command.c @@ -993,11 +993,8 @@ virBhyveGetBootDisk(virDomainDefPtr def) virDomainDiskDefPtr match = NULL; int boot_dev = -1;
- if (def->ndisks < 1) { - virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", - _("Domain should have at least one disk defined")); + if (def->ndisks == 0) return NULL; - }
This doesn't look like correct logic to me. Most code paths which return NULL will report an error, so having one code path which doesn't is pretty unusual practice. Also the caller of this method still assumes a NULL return value indicates an error AFAICS.
if (def->os.nBootDevs > 1) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", -- 2.24.1
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

PCI Alternative Routing-ID Interpretation (ARI) capable devices use an implied Device Number of 0 and repurpose those bits to allow for 256 Function Numbers instead of 8. An example of such a device is the Chelsio T580-LP-CR, which uses the additional function numbers to provide Virtual Function (VF) features. Below is an excerpt from the `pciconf -lvc` output for this device when a few VFs are configured: ppt7@pci0:5:0:21: class=0x020000 card=0x00001425 chip=0x58101425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T580-LP-CR Unified Wire Ethernet Controller [VF]' class = network subclass = ethernet cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR NS link x0(x8) speed 0.0(8.0) ASPM disabled(L0s/L1) cap 11[b0] = MSI-X supports 8 messages Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] = MSI supports 32 messages, 64 bit, vector masks ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 000e[140] = ARI 1 ecap 0017[150] = TPH Requester 1 Of note: the Bus/Device/Function 5/0/21 and the ARI ecap. Attempting to pass this through to a VM as a PCI device produces a validation error due to the function number being out of range. To enable use of devices that use ARI, relax the schema and validity check for PCI addresses to permit function numbers up to 255. Update a few tests to give coverage. Ref: https://pcisig.com/sites/default/files/specification_documents/ECN-alt-rid-i... Signed-off-by: Ryan Moeller <ryan@iXsystems.com> --- docs/schemas/basictypes.rng | 10 +--------- src/util/virpci.c | 4 ++-- tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args | 2 +- tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml | 2 +- tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args | 2 +- tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml | 2 +- tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml | 2 +- tests/qemuxml2argvdata/pci-function-invalid.xml | 2 +- 8 files changed, 9 insertions(+), 17 deletions(-) diff --git a/docs/schemas/basictypes.rng b/docs/schemas/basictypes.rng index 81465273c8..03ce3f3be3 100644 --- a/docs/schemas/basictypes.rng +++ b/docs/schemas/basictypes.rng @@ -341,15 +341,7 @@ </choice> </define> <define name="pciFunc"> - <choice> - <data type="string"> - <param name="pattern">(0x)?[0-7]</param> - </data> - <data type="int"> - <param name="minInclusive">0</param> - <param name="maxInclusive">7</param> - </data> - </choice> + <ref name="uint8"/> </define> <define name='wwn'> diff --git a/src/util/virpci.c b/src/util/virpci.c index 0b1222373e..d60cf5eff7 100644 --- a/src/util/virpci.c +++ b/src/util/virpci.c @@ -1302,11 +1302,11 @@ virPCIDeviceAddressIsValid(virPCIDeviceAddressPtr addr, addr->slot); return false; } - if (addr->function > 7) { + if (addr->function > 255) { if (report) virReportError(VIR_ERR_XML_ERROR, _("Invalid PCI address function=0x%x, " - "must be <= 7"), + "must be <= 255"), addr->function); return false; } diff --git a/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args index 697bafd642..726dade59e 100644 --- a/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args +++ b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.args @@ -5,4 +5,4 @@ -P \ -S \ -s 0:0,hostbridge \ --s 0:1:0,passthru,5/0/7 bhyve +-s 0:1:0,passthru,5/0/9 bhyve diff --git a/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml index af99279448..a8d7a574af 100644 --- a/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml +++ b/tests/bhyveargv2xmldata/bhyveargv2xml-passthru.xml @@ -18,7 +18,7 @@ <hostdev mode='subsystem' type='pci' managed='no'> <driver name='vmm'/> <source> - <address domain='0x0000' bus='0x05' slot='0x00' function='0x7'/> + <address domain='0x0000' bus='0x05' slot='0x00' function='0x9'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </hostdev> diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args index c268da957c..afa99344af 100644 --- a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.args @@ -7,5 +7,5 @@ -P \ -s 0:0,hostbridge \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE-devel.fd \ --s 0:3:0,passthru,5/0/7 \ +-s 0:3:0,passthru,5/0/9 \ -s 1,lpc bhyve diff --git a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml index ba0744f35d..e8b3a4064f 100644 --- a/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml +++ b/tests/bhyvexml2argvdata/bhyvexml2argv-passthru.xml @@ -14,7 +14,7 @@ <hostdev mode='subsystem' type='pci' managed='no'> <driver name='vmm'/> <source> - <address domain='0x0000' bus='0x05' slot='0x00' function='0x7'/> + <address domain='0x0000' bus='0x05' slot='0x00' function='0x9'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> diff --git a/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml b/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml index 0313fa0dfa..7f96831df9 100644 --- a/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml +++ b/tests/bhyvexml2xmloutdata/bhyvexml2xmlout-passthru.xml @@ -21,7 +21,7 @@ <hostdev mode='subsystem' type='pci' managed='no'> <driver name='vmm'/> <source> - <address domain='0x0000' bus='0x05' slot='0x00' function='0x7'/> + <address domain='0x0000' bus='0x05' slot='0x00' function='0x9'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> diff --git a/tests/qemuxml2argvdata/pci-function-invalid.xml b/tests/qemuxml2argvdata/pci-function-invalid.xml index 5642809ec0..eced7b3fa6 100644 --- a/tests/qemuxml2argvdata/pci-function-invalid.xml +++ b/tests/qemuxml2argvdata/pci-function-invalid.xml @@ -26,7 +26,7 @@ <interface type='user'> <mac address='00:11:22:33:44:55'/> <model type='rtl8139'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x8'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x100'/> </interface> <memballoon model='none'/> </devices> -- 2.24.1

On Mon, Feb 24, 2020 at 01:46:12AM -0500, Ryan Moeller wrote:
Rebased and updated from previous patch set to address feedback:
* Tried to match local convention for subjects where obvious * Split patch 01 into two patches, with updated messages * Use g_autofree to fix use after free in conf/virnetworkobj * Add missing newline in one of the tests args files * Fix failing schema tests after schema change
gmake check now reports no failing tests on FreeBSD for each patch.
Ryan Moeller (12): bhyve: process: remove unneeded header conf: fix use after free bhyve: process: don't bother seeking to end of log bhyve: monitor: Make bhyveMonitor a virClass bhyve: monitor: refactor register/unregister bhyve: add hooks bhyve: add reboot support bhyve: command: refactor virBhyveProcessBuildBhyveCmd bhyve: parse_command: slot,bus,func -> bus,slot,func
Thanks for the contribution, I've pushed these patches now since they were independant of the remaining three.
add hostdev handling for bhyve bhyve: command: enable booting from hostdevs Allow PCI functions up to 255 for PCI ARI
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Awesome, thank you! I'll try to address the remaining feedback when I get a chance. This work isn't exactly my primary job function so it's been more of a nights and weekends labor of love kind of thing. Thanks again for taking the time to review and push these patches! On Mon, Feb 24, 2020 at 1:07 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Feb 24, 2020 at 01:46:12AM -0500, Ryan Moeller wrote:
Rebased and updated from previous patch set to address feedback:
* Tried to match local convention for subjects where obvious * Split patch 01 into two patches, with updated messages * Use g_autofree to fix use after free in conf/virnetworkobj * Add missing newline in one of the tests args files * Fix failing schema tests after schema change
gmake check now reports no failing tests on FreeBSD for each patch.
Ryan Moeller (12): bhyve: process: remove unneeded header conf: fix use after free bhyve: process: don't bother seeking to end of log bhyve: monitor: Make bhyveMonitor a virClass bhyve: monitor: refactor register/unregister bhyve: add hooks bhyve: add reboot support bhyve: command: refactor virBhyveProcessBuildBhyveCmd bhyve: parse_command: slot,bus,func -> bus,slot,func
Thanks for the contribution, I've pushed these patches now since they were independant of the remaining three.
add hostdev handling for bhyve bhyve: command: enable booting from hostdevs Allow PCI functions up to 255 for PCI ARI
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
-- Ryan Moeller iXsystems, Inc. OS Developer Email: ryan@iXsystems.com
participants (2)
-
Daniel P. Berrangé
-
Ryan Moeller