[libvirt PATCH 0/3] nodedev: fix device reset/detach/reattach with split daemons
The virNodeDevice{Reset,ReAttach,Dettach} methods are a little unusual because they are registered by the virt driver, but use the virNodeDevicePtr object associated with the nodedev driver. In the split daemon world, we tried to dispatch them to the virnodedevd daemon which has no impl registered. Dispatching them to the virt driver has some quirks, we need to ensure we have two virNodeDevicePtr instances, once for the virConnectPtr of the virt driver and one for the virConnectPtr of the nodedev driver. Daniel P. Berrangé (3): rpc: fix dispatch for node device APIs for virt drivers rpc: avoid name lookup when dispatching node device APIs qemu: lookup node device against nodedev driver before getting XML src/libxl/libxl_driver.c | 63 +++++++++++++++++++++++++++-- src/qemu/qemu_driver.c | 63 +++++++++++++++++++++++++++-- src/remote/remote_daemon_dispatch.c | 7 ++++ src/rpc/gendispatch.pl | 10 ++++- 4 files changed, 135 insertions(+), 8 deletions(-) -- 2.24.1
Despite their names, the following APIs: virNodeDeviceDettach virNodeDeviceDetachFlags virNodeDeviceReAttach virNodeDeviceReset are all handled by the virt drivers, not the node device driver. A bug in the RPC generator meant that these APIs were sent to the nodedev driver for handling. This caused breakage with the split daemons, since nothing was available to process them. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/rpc/gendispatch.pl | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/rpc/gendispatch.pl b/src/rpc/gendispatch.pl index 987a136566..c140ed712c 100755 --- a/src/rpc/gendispatch.pl +++ b/src/rpc/gendispatch.pl @@ -171,7 +171,13 @@ sub get_conn_method { if ($proc =~ /Connect.*Network/) { return "remoteGetNetworkConn"; } - if ($proc =~ /Node.*Device/) { + # Carefully whitelist a few APIs with NodeDevice name + # prefix which actually get handled by the virt drivers + if ($proc =~ /Node.*Device/ && + !($proc =~ /NodeDeviceReset/ || + $proc =~ /NodeDeviceReAttach/ || + $proc =~ /NodeDeviceDettach/ || + $proc =~ /NodeDeviceDetachFlags/)) { return "remoteGetNodeDevConn"; } if ($proc =~ /Connect.*NWFilter/) { -- 2.24.1
The node device APIs are a little unusual because we don't use a "remote_nonnull_node_device" object on the wire, instead we just have a "remote_string" for the device name. This meant dispatcher code generation needed special cases. In doing so we mistakenly used the virNodeDeviceLookupByName() API which gets dispatched into the driver, instead of get_nonnull_node_device() which directly populates a virNodeDevicePtr object. This wasn't a problem with monolithic libvirtd, as the virNodeDeviceLookupByName() API call was trivially satisfied by the registered driver, albeit with an extra (undesirable) authentication check. With the split daemons, the call to virNodeDeviceLookupByName() fails in virtqemud, because the node device driver obviously doesn't exist in that daemon. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/remote/remote_daemon_dispatch.c | 7 +++++++ src/rpc/gendispatch.pl | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/src/remote/remote_daemon_dispatch.c b/src/remote/remote_daemon_dispatch.c index 2741a32f63..226049fed6 100644 --- a/src/remote/remote_daemon_dispatch.c +++ b/src/remote/remote_daemon_dispatch.c @@ -95,6 +95,7 @@ static virNWFilterBindingPtr get_nonnull_nwfilter_binding(virConnectPtr conn, re static virDomainCheckpointPtr get_nonnull_domain_checkpoint(virDomainPtr dom, remote_nonnull_domain_checkpoint checkpoint); static virDomainSnapshotPtr get_nonnull_domain_snapshot(virDomainPtr dom, remote_nonnull_domain_snapshot snapshot); static virNodeDevicePtr get_nonnull_node_device(virConnectPtr conn, remote_nonnull_node_device dev); +static virNodeDevicePtr get_nonnull_node_device_name(virConnectPtr conn, remote_nonnull_string devdev); static void make_nonnull_domain(remote_nonnull_domain *dom_dst, virDomainPtr dom_src); static void make_nonnull_network(remote_nonnull_network *net_dst, virNetworkPtr net_src); static void make_nonnull_network_port(remote_nonnull_network_port *port_dst, virNetworkPortPtr port_src); @@ -7291,6 +7292,12 @@ get_nonnull_node_device(virConnectPtr conn, remote_nonnull_node_device dev) return virGetNodeDevice(conn, dev.name); } +static virNodeDevicePtr +get_nonnull_node_device_name(virConnectPtr conn, remote_nonnull_string devname) +{ + return virGetNodeDevice(conn, devname); +} + static void make_nonnull_domain(remote_nonnull_domain *dom_dst, virDomainPtr dom_src) { diff --git a/src/rpc/gendispatch.pl b/src/rpc/gendispatch.pl index c140ed712c..0b2ae59910 100755 --- a/src/rpc/gendispatch.pl +++ b/src/rpc/gendispatch.pl @@ -571,7 +571,7 @@ elsif ($mode eq "server") { $has_node_device = 1; push(@vars_list, "virNodeDevicePtr dev = NULL"); push(@getters_list, - " if (!(dev = virNodeDeviceLookupByName($conn_var, args->name)))\n" . + " if (!(dev = get_nonnull_node_device_name($conn_var, args->name)))\n" . " goto cleanup;\n"); push(@args_list, "dev"); push(@free_list, -- 2.24.1
On 10. 3. 2020 19:13, Daniel P. Berrangé wrote:
The node device APIs are a little unusual because we don't use a "remote_nonnull_node_device" object on the wire, instead we just have a "remote_string" for the device name. This meant dispatcher code generation needed special cases. In doing so we mistakenly used the virNodeDeviceLookupByName() API which gets dispatched into the driver, instead of get_nonnull_node_device() which directly populates a virNodeDevicePtr object.
This wasn't a problem with monolithic libvirtd, as the virNodeDeviceLookupByName() API call was trivially satisfied by the registered driver, albeit with an extra (undesirable) authentication check. With the split daemons, the call to virNodeDeviceLookupByName() fails in virtqemud, because the node device driver obviously doesn't exist in that daemon.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/remote/remote_daemon_dispatch.c | 7 +++++++ src/rpc/gendispatch.pl | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/remote/remote_daemon_dispatch.c b/src/remote/remote_daemon_dispatch.c index 2741a32f63..226049fed6 100644 --- a/src/remote/remote_daemon_dispatch.c +++ b/src/remote/remote_daemon_dispatch.c @@ -95,6 +95,7 @@ static virNWFilterBindingPtr get_nonnull_nwfilter_binding(virConnectPtr conn, re static virDomainCheckpointPtr get_nonnull_domain_checkpoint(virDomainPtr dom, remote_nonnull_domain_checkpoint checkpoint); static virDomainSnapshotPtr get_nonnull_domain_snapshot(virDomainPtr dom, remote_nonnull_domain_snapshot snapshot); static virNodeDevicePtr get_nonnull_node_device(virConnectPtr conn, remote_nonnull_node_device dev); +static virNodeDevicePtr get_nonnull_node_device_name(virConnectPtr conn, remote_nonnull_string devdev); static void make_nonnull_domain(remote_nonnull_domain *dom_dst, virDomainPtr dom_src); static void make_nonnull_network(remote_nonnull_network *net_dst, virNetworkPtr net_src); static void make_nonnull_network_port(remote_nonnull_network_port *port_dst, virNetworkPortPtr port_src); @@ -7291,6 +7292,12 @@ get_nonnull_node_device(virConnectPtr conn, remote_nonnull_node_device dev) return virGetNodeDevice(conn, dev.name); }
+static virNodeDevicePtr +get_nonnull_node_device_name(virConnectPtr conn, remote_nonnull_string devname)
s/devname/dev/ because syntax-check suggests that @devname is a function on FreeBSD.
+{ + return virGetNodeDevice(conn, devname); +} + static void make_nonnull_domain(remote_nonnull_domain *dom_dst, virDomainPtr dom_src) { diff --git a/src/rpc/gendispatch.pl b/src/rpc/gendispatch.pl index c140ed712c..0b2ae59910 100755 --- a/src/rpc/gendispatch.pl +++ b/src/rpc/gendispatch.pl @@ -571,7 +571,7 @@ elsif ($mode eq "server") { $has_node_device = 1; push(@vars_list, "virNodeDevicePtr dev = NULL"); push(@getters_list, - " if (!(dev = virNodeDeviceLookupByName($conn_var, args->name)))\n" . + " if (!(dev = get_nonnull_node_device_name($conn_var, args->name)))\n" . " goto cleanup;\n"); push(@args_list, "dev"); push(@free_list,
Michal
Some of the node device APIs are a little odd because they accept a virNodeDevicePtr object but are still implemented by the virt drivers. The first thing the virt drivers need to do is get the XML config associated with the node device, and that means talking to the node device driver. This worked previously because with monolithic libvirtd, both the virt driver and node device driver were in the same daemon and thus a single virConnectPtr can talk to both drivers. With the split daemon world though, the virNodeDevicePtr passed into the APIs is associated with the QEMU driver virConnectPtr, which has no ability to invoke APIs against the node device driver. We must thus get a duplicate virNodeDevicePtr object which is associated with a virConnectPtr for the node device driver. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/libxl/libxl_driver.c | 63 ++++++++++++++++++++++++++++++++++++++-- src/qemu/qemu_driver.c | 63 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 120 insertions(+), 6 deletions(-) diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index f2387e2a20..eca45da097 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -5787,10 +5787,23 @@ libxlNodeDeviceDetachFlags(virNodeDevicePtr dev, char *xml = NULL; libxlDriverPrivatePtr driver = dev->conn->privateData; virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL; virCheckFlags(0, -1); - xml = virNodeDeviceGetXMLDesc(dev, 0); + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; + + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev)))) + goto cleanup; + + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup; @@ -5798,6 +5811,8 @@ libxlNodeDeviceDetachFlags(virNodeDevicePtr dev, if (!def) goto cleanup; + /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceDetachFlagsEnsureACL(dev->conn, def) < 0) goto cleanup; @@ -5823,6 +5838,10 @@ libxlNodeDeviceDetachFlags(virNodeDevicePtr dev, cleanup: virPCIDeviceFree(pci); virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev); + if (nodeconn) + virConnectClose(nodeconn); VIR_FREE(xml); return ret; } @@ -5843,8 +5862,21 @@ libxlNodeDeviceReAttach(virNodeDevicePtr dev) char *xml = NULL; libxlDriverPrivatePtr driver = dev->conn->privateData; virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL; + + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; - xml = virNodeDeviceGetXMLDesc(dev, 0); + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev)))) + goto cleanup; + + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup; @@ -5852,6 +5884,8 @@ libxlNodeDeviceReAttach(virNodeDevicePtr dev) if (!def) goto cleanup; + /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceReAttachEnsureACL(dev->conn, def) < 0) goto cleanup; @@ -5870,6 +5904,10 @@ libxlNodeDeviceReAttach(virNodeDevicePtr dev) cleanup: virPCIDeviceFree(pci); virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev); + if (nodeconn) + virConnectClose(nodeconn); VIR_FREE(xml); return ret; } @@ -5884,8 +5922,21 @@ libxlNodeDeviceReset(virNodeDevicePtr dev) char *xml = NULL; libxlDriverPrivatePtr driver = dev->conn->privateData; virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL; + + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; + + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev)))) + goto cleanup; - xml = virNodeDeviceGetXMLDesc(dev, 0); + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup; @@ -5893,6 +5944,8 @@ libxlNodeDeviceReset(virNodeDevicePtr dev) if (!def) goto cleanup; + /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceResetEnsureACL(dev->conn, def) < 0) goto cleanup; @@ -5911,6 +5964,10 @@ libxlNodeDeviceReset(virNodeDevicePtr dev) cleanup: virPCIDeviceFree(pci); virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev); + if (nodeconn) + virConnectClose(nodeconn); VIR_FREE(xml); return ret; } diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index cd761f87b5..d8f264ba7f 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12965,10 +12965,23 @@ qemuNodeDeviceDetachFlags(virNodeDevicePtr dev, g_autofree char *xml = NULL; bool vfio = qemuHostdevHostSupportsPassthroughVFIO(); virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL; virCheckFlags(0, -1); - xml = virNodeDeviceGetXMLDesc(dev, 0); + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; + + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev)))) + goto cleanup; + + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup; @@ -12976,6 +12989,8 @@ qemuNodeDeviceDetachFlags(virNodeDevicePtr dev, if (!def) goto cleanup; + /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceDetachFlagsEnsureACL(dev->conn, def) < 0) goto cleanup; @@ -13012,6 +13027,10 @@ qemuNodeDeviceDetachFlags(virNodeDevicePtr dev, cleanup: virPCIDeviceFree(pci); virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev); + if (nodeconn) + virConnectClose(nodeconn); return ret; } @@ -13031,8 +13050,21 @@ qemuNodeDeviceReAttach(virNodeDevicePtr dev) virNodeDeviceDefPtr def = NULL; g_autofree char *xml = NULL; virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL; + + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; - xml = virNodeDeviceGetXMLDesc(dev, 0); + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev)))) + goto cleanup; + + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup; @@ -13040,6 +13072,8 @@ qemuNodeDeviceReAttach(virNodeDevicePtr dev) if (!def) goto cleanup; + /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceReAttachEnsureACL(dev->conn, def) < 0) goto cleanup; @@ -13055,6 +13089,10 @@ qemuNodeDeviceReAttach(virNodeDevicePtr dev) virPCIDeviceFree(pci); cleanup: virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev); + if (nodeconn) + virConnectClose(nodeconn); return ret; } @@ -13068,8 +13106,21 @@ qemuNodeDeviceReset(virNodeDevicePtr dev) virNodeDeviceDefPtr def = NULL; g_autofree char *xml = NULL; virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL; + + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; + + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev)))) + goto cleanup; - xml = virNodeDeviceGetXMLDesc(dev, 0); + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup; @@ -13077,6 +13128,8 @@ qemuNodeDeviceReset(virNodeDevicePtr dev) if (!def) goto cleanup; + /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceResetEnsureACL(dev->conn, def) < 0) goto cleanup; @@ -13092,6 +13145,10 @@ qemuNodeDeviceReset(virNodeDevicePtr dev) virPCIDeviceFree(pci); cleanup: virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev); + if (nodeconn) + virConnectClose(nodeconn); return ret; } -- 2.24.1
On 10. 3. 2020 19:13, Daniel P. Berrangé wrote:
Some of the node device APIs are a little odd because they accept a virNodeDevicePtr object but are still implemented by the virt drivers. The first thing the virt drivers need to do is get the XML config associated with the node device, and that means talking to the node device driver.
This worked previously because with monolithic libvirtd, both the virt driver and node device driver were in the same daemon and thus a single virConnectPtr can talk to both drivers.
With the split daemon world though, the virNodeDevicePtr passed into the APIs is associated with the QEMU driver virConnectPtr, which has no ability to invoke APIs against the node device driver. We must thus get a duplicate virNodeDevicePtr object which is associated with a virConnectPtr for the node device driver.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- src/libxl/libxl_driver.c | 63 ++++++++++++++++++++++++++++++++++++++-- src/qemu/qemu_driver.c | 63 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 120 insertions(+), 6 deletions(-)
diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index f2387e2a20..eca45da097 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -5787,10 +5787,23 @@ libxlNodeDeviceDetachFlags(virNodeDevicePtr dev, char *xml = NULL; libxlDriverPrivatePtr driver = dev->conn->privateData; virHostdevManagerPtr hostdev_mgr = driver->hostdevMgr; + virConnectPtr nodeconn = NULL; + virNodeDevicePtr nodedev = NULL;
virCheckFlags(0, -1);
- xml = virNodeDeviceGetXMLDesc(dev, 0); + if (!(nodeconn = virGetConnectNodeDev())) + goto cleanup; + + /* 'dev' is associated with the QEMU virConnectPtr, + * so for split daemons, we need to get a copy that + * is associated with the virnodedevd daemon. + */ + if (!(nodedev = virNodeDeviceLookupByName( + nodeconn, virNodeDeviceGetName(dev))))
No need to split this line IMO. 85 characters long line is not that bad.
+ goto cleanup; + + xml = virNodeDeviceGetXMLDesc(nodedev, 0); if (!xml) goto cleanup;
@@ -5798,6 +5811,8 @@ libxlNodeDeviceDetachFlags(virNodeDevicePtr dev, if (!def) goto cleanup;
+ /* ACL check must happen against original 'dev', + * not the new 'nodedev' we acquired */ if (virNodeDeviceDetachFlagsEnsureACL(dev->conn, def) < 0) goto cleanup;
@@ -5823,6 +5838,10 @@ libxlNodeDeviceDetachFlags(virNodeDevicePtr dev, cleanup: virPCIDeviceFree(pci); virNodeDeviceDefFree(def); + if (nodedev) + virNodeDeviceFree(nodedev);
s/virNodeDeviceFree/virObjectUnref/ as we don't really need to call the public API.
+ if (nodeconn) + virConnectClose(nodeconn);
And seems like we are preferring virObjectUnref() over virConnectClose() too.
VIR_FREE(xml); return ret; }
Michal
On 10. 3. 2020 19:13, Daniel P. Berrangé wrote:
The virNodeDevice{Reset,ReAttach,Dettach} methods are a little unusual because they are registered by the virt driver, but use the virNodeDevicePtr object associated with the nodedev driver.
In the split daemon world, we tried to dispatch them to the virnodedevd daemon which has no impl registered. Dispatching them to the virt driver has some quirks, we need to ensure we have two virNodeDevicePtr instances, once for the virConnectPtr of the virt driver and one for the virConnectPtr of the nodedev driver.
Daniel P. Berrangé (3): rpc: fix dispatch for node device APIs for virt drivers rpc: avoid name lookup when dispatching node device APIs qemu: lookup node device against nodedev driver before getting XML
src/libxl/libxl_driver.c | 63 +++++++++++++++++++++++++++-- src/qemu/qemu_driver.c | 63 +++++++++++++++++++++++++++-- src/remote/remote_daemon_dispatch.c | 7 ++++ src/rpc/gendispatch.pl | 10 ++++- 4 files changed, 135 insertions(+), 8 deletions(-)
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal
participants (2)
-
Daniel P. Berrangé -
Michal Prívozník