[libvirt PATCH] nodedev: switch to udev 'bind' events
Rather than listening to 'add' udev events, listen for 'bind' events instead. When we get an 'add' event, the sysfs tree for the device is often not ready yet. In that case we sleep in a loop until the sysfs tree appears, or give up after a timeout. udev added the 'bind' event to give userspace a signal that indicated when driver-specific attributes were available to be used. In other words, the sysfs tree *should* be ready and usable at this point. But just to be safe, we'll leave the wait loop in the code to handle corner cases, with the hope that it'll never be used. The udev 'bind' event was added in kernel 4.14 and the oldest platform we support has kernel 4.18, so it should be safe to make this change. Previous discussion on the mailing list: https://listman.redhat.com/archives/libvir-list/2022-August/233933.html Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> --- src/node_device/node_device_udev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 07c10f0d88..781a8b32a6 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1759,7 +1759,7 @@ udevHandleOneDevice(struct udev_device *device) VIR_DEBUG("udev action: '%s': %s", action, udev_device_get_syspath(device)); - if (STREQ(action, "add") || STREQ(action, "change")) + if (STREQ(action, "bind") || STREQ(action, "change")) return udevAddOneDevice(device); if (STREQ(action, "remove")) -- 2.37.2
On 9/20/22 3:23 PM, Jonathon Jongsma wrote:
Rather than listening to 'add' udev events, listen for 'bind' events instead. When we get an 'add' event, the sysfs tree for the device is often not ready yet. In that case we sleep in a loop until the sysfs tree appears, or give up after a timeout.
udev added the 'bind' event to give userspace a signal that indicated when driver-specific attributes were available to be used. In other words, the sysfs tree *should* be ready and usable at this point. But just to be safe, we'll leave the wait loop in the code to handle corner cases, with the hope that it'll never be used.
The udev 'bind' event was added in kernel 4.14 and the oldest platform we support has kernel 4.18, so it should be safe to make this change.
Previous discussion on the mailing list: https://listman.redhat.com/archives/libvir-list/2022-August/233933.html
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com>
Reviewed-by: Laine Stump <laine@redhat.com> (I'm giving this based on the assumption that you've tried it out with a few examples of adding devices and they properly show up in virsh nodedev-list, etc. There is always the chance may be some odd case where the bind event isn't generated when it should be, but we're never going to find out by just talking about it :-))
--- src/node_device/node_device_udev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 07c10f0d88..781a8b32a6 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1759,7 +1759,7 @@ udevHandleOneDevice(struct udev_device *device)
VIR_DEBUG("udev action: '%s': %s", action, udev_device_get_syspath(device));
- if (STREQ(action, "add") || STREQ(action, "change")) + if (STREQ(action, "bind") || STREQ(action, "change")) return udevAddOneDevice(device);
if (STREQ(action, "remove"))
On Tue, Sep 20, 2022 at 02:23:23PM -0500, Jonathon Jongsma wrote:
Rather than listening to 'add' udev events, listen for 'bind' events instead. When we get an 'add' event, the sysfs tree for the device is often not ready yet. In that case we sleep in a loop until the sysfs tree appears, or give up after a timeout.
udev added the 'bind' event to give userspace a signal that indicated when driver-specific attributes were available to be used. In other words, the sysfs tree *should* be ready and usable at this point. But just to be safe, we'll leave the wait loop in the code to handle corner cases, with the hope that it'll never be used.
The udev 'bind' event was added in kernel 4.14 and the oldest platform we support has kernel 4.18, so it should be safe to make this change.
Previous discussion on the mailing list: https://listman.redhat.com/archives/libvir-list/2022-August/233933.html
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> ---
Unfortunately I don't have an mdev-configured machine at hand anymore to try this out, but based on the trivial change and assuming you've at least tried the patch, you can have my: Reviewed-by: Erik Skultety <eskultet@redhat.com>
On 9/22/22 3:59 AM, Erik Skultety wrote:
On Tue, Sep 20, 2022 at 02:23:23PM -0500, Jonathon Jongsma wrote:
Rather than listening to 'add' udev events, listen for 'bind' events instead. When we get an 'add' event, the sysfs tree for the device is often not ready yet. In that case we sleep in a loop until the sysfs tree appears, or give up after a timeout.
udev added the 'bind' event to give userspace a signal that indicated when driver-specific attributes were available to be used. In other words, the sysfs tree *should* be ready and usable at this point. But just to be safe, we'll leave the wait loop in the code to handle corner cases, with the hope that it'll never be used.
The udev 'bind' event was added in kernel 4.14 and the oldest platform we support has kernel 4.18, so it should be safe to make this change.
Previous discussion on the mailing list: https://listman.redhat.com/archives/libvir-list/2022-August/233933.html
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> ---
Unfortunately I don't have an mdev-configured machine at hand anymore to try this out, but based on the trivial change and assuming you've at least tried the patch, you can have my:
Reviewed-by: Erik Skultety <eskultet@redhat.com>
I have tried it with several devices including usb and mdev. But I'm no longer sure whether this is a safe change. For example, imagine I have a device configured to not load a driver (e.g. perhaps something like `driverctl set-override $DEVICE none`). If this device is hotplugged, we should get a udev 'add' event but no 'bind' event. So, previously this driverless device would have showed up in `virsh nodedev-list`, but after this change it will not. Also note that this creates a difference between startup and hotplug. At libvirt startup we enumerate all devices and add them to our nodedev list. The enumerated devices will include devices both with bound drivers and those without. But if we only listen to the 'bind' event, hotplugged devices will need drivers bound to be added to the list. So if you hotplug a driverless device, it will not show up in the nodedev list until libvirt is restarted. Note that the above description is based only on my analysis of the code, not testing. I'm inclined to withdraw this patch. Jonathon
On 9/22/22 12:41 PM, Jonathon Jongsma wrote:
On 9/22/22 3:59 AM, Erik Skultety wrote:
On Tue, Sep 20, 2022 at 02:23:23PM -0500, Jonathon Jongsma wrote:
Rather than listening to 'add' udev events, listen for 'bind' events instead. When we get an 'add' event, the sysfs tree for the device is often not ready yet. In that case we sleep in a loop until the sysfs tree appears, or give up after a timeout.
udev added the 'bind' event to give userspace a signal that indicated when driver-specific attributes were available to be used. In other words, the sysfs tree *should* be ready and usable at this point. But just to be safe, we'll leave the wait loop in the code to handle corner cases, with the hope that it'll never be used.
The udev 'bind' event was added in kernel 4.14 and the oldest platform we support has kernel 4.18, so it should be safe to make this change.
Previous discussion on the mailing list: https://listman.redhat.com/archives/libvir-list/2022-August/233933.html
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> ---
Unfortunately I don't have an mdev-configured machine at hand anymore to try this out, but based on the trivial change and assuming you've at least tried the patch, you can have my:
Reviewed-by: Erik Skultety <eskultet@redhat.com>
I have tried it with several devices including usb and mdev.
But I'm no longer sure whether this is a safe change. For example, imagine I have a device configured to not load a driver (e.g. perhaps something like `driverctl set-override $DEVICE none`). If this device is hotplugged, we should get a udev 'add' event but no 'bind' event. So, previously this driverless device would have showed up in `virsh nodedev-list`, but after this change it will not.
Also note that this creates a difference between startup and hotplug. At libvirt startup we enumerate all devices and add them to our nodedev list. The enumerated devices will include devices both with bound drivers and those without. But if we only listen to the 'bind' event, hotplugged devices will need drivers bound to be added to the list. So if you hotplug a driverless device, it will not show up in the nodedev list until libvirt is restarted.
Hmm. Good catch - this is one of those things I would have just ended up finding out later on :-). Maybe we need to listen for both events, and as each one is received, we populate what we can of the entry in the device list? (On one hand, there's nothing that libvirt can directly do with a device that has no binding to a driver; on the other hand, it's not just libvirt looking at the list of devices - someone may specifically want their devices to have no driver when the host boots, but then wants to discover them from the nodedev list and bind them to vfio-pci at runtime.)
Note that the above description is based only on my analysis of the code, not testing.
I'm inclined to withdraw this patch.
Jonathon
participants (3)
-
Erik Skultety -
Jonathon Jongsma -
Laine Stump