On 4/16/20 5:57 PM, Mark Asselstine wrote:

The udev monitor thread "udevEventHandleThread()" will lag the actual/real view of devices in sysfs as it serially processes udev monitor events. So for instance if you were to run the following cmd to create a new veth pair and rename one of the veth endpoints

you might see the following monitor events and real world that looks like

time | create v0 sysfs entry wake udevEventHandleThread | create v1 sysfs entry udev_monitor_receive_device(v1-add) | move v0 sysfs to v2 udevHandleOneDevice(v1) | udev_monitor_receive_device(v0-add) | udevHandleOneDevice(v0) | <--- error msgs in virNetDevGetLinkInfo() udev_monitor_receive_device(v2-move) | as v0 no longer exists udevHandleOneDevice(v2) | \/

As you can see the changes in sysfs can take place well before we get to act on the events in the udevEventHandleThread(), so by the time we get around to processing the v0 add event, the sysfs entry has been moved to v2.

To work around this we check if the sysfs entry is valid before attempting to read it and don't bother trying to read link info if not. This is safe since we will never read sysfs entries earlier than it existing, ie. if the entry is not there it has either been removed in the time since we enumerated the device or something bigger is busted, in either case, no sysfs entry, no link info. In the case described above we will eventually get the link info as we work through the queue of monitor events and get to the 'move' event.

https://bugzilla.redhat.com/show_bug.cgi?id=1557902

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> --- src/util/virnetdev.c | 11 +++++++++++ 1 file changed, 11 insertions(+)

diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index b465bdac2e..bf256cbe35 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -2438,6 +2438,17 @@ virNetDevGetLinkInfo(const char *ifname, if (virNetDevSysfsFile(&path, ifname, "operstate") < 0) goto cleanup;

+ /* The device may have been removed or moved by the time we got here. + * Obviously attempting to get LinkInfo on a no longer existing device + * is useless, so stop processing. If we got here via the udev monitor + * a remove or move event will follow and we will be able to get valid + * LinkInfo at that time */ + if (!virFileExists(path)) { + VIR_WARN("The interface '%s' was removed before we could query it.", + ifname);

This is not aligned.

+ goto cleanup; + }

But I have another idea to consider. We could return a different error value here (say -2) and then have the caller decide what to do with it (e.g. print the warning). But this is still inherently racy. The "move" can happen just after we've checked for the @path existence and before this virFileReadAll() below. But I guess this is still better than nothing. Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal

On 4/20/20 3:09 PM, Mark Asselstine wrote:

On Monday, April 20, 2020 6:11:25 A.M. EDT Michal Privoznik wrote:

...

On 4/16/20 5:57 PM, Mark Asselstine wrote:

...

The udev monitor thread "udevEventHandleThread()" will lag the actual/real view of devices in sysfs as it serially processes udev monitor events. So for instance if you were to run the following cmd to create a new veth pair and rename one of the veth endpoints

you might see the following monitor events and real world that looks like

time | | create v0 sysfs entry

wake udevEventHandleThread | create v1 sysfs entry udev_monitor_receive_device(v1-add) | move v0 sysfs to v2 udevHandleOneDevice(v1) | udev_monitor_receive_device(v0-add) | udevHandleOneDevice(v0) | <--- error msgs in virNetDevGetLinkInfo() udev_monitor_receive_device(v2-move) | as v0 no longer exists udevHandleOneDevice(v2) |

\/

As you can see the changes in sysfs can take place well before we get to act on the events in the udevEventHandleThread(), so by the time we get around to processing the v0 add event, the sysfs entry has been moved to v2.

To work around this we check if the sysfs entry is valid before attempting to read it and don't bother trying to read link info if not. This is safe since we will never read sysfs entries earlier than it existing, ie. if the entry is not there it has either been removed in the time since we enumerated the device or something bigger is busted, in either case, no sysfs entry, no link info. In the case described above we will eventually get the link info as we work through the queue of monitor events and get to the 'move' event.

https://bugzilla.redhat.com/show_bug.cgi?id=1557902

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> ---

src/util/virnetdev.c | 11 +++++++++++ 1 file changed, 11 insertions(+)

diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index b465bdac2e..bf256cbe35 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -2438,6 +2438,17 @@ virNetDevGetLinkInfo(const char *ifname,

if (virNetDevSysfsFile(&path, ifname, "operstate") < 0)

goto cleanup;

+ /* The device may have been removed or moved by the time we got here. + * Obviously attempting to get LinkInfo on a no longer existing device + * is useless, so stop processing. If we got here via the udev monitor + * a remove or move event will follow and we will be able to get valid + * LinkInfo at that time */ + if (!virFileExists(path)) { + VIR_WARN("The interface '%s' was removed before we could query it.", + ifname);

This is not aligned.

Should I send a V2 or can this be correct when merged?

No need, I've fixed it (also changed goto cleanup to return, since I've merged a patch that made 'cleanup' label go away this morning) and pushed.

...

...

+ goto cleanup; + }

But I have another idea to consider. We could return a different error value here (say -2) and then have the caller decide what to do with it (e.g. print the warning).

I had thought about propagating the error so the caller could possibly wrap the warning message in more context but in the end I decided to call it out at the issue site avoiding having each caller essentially reproducing essentially the same code.

Yeah, what convinced me was that non-Linux version of this function simply prints a debug message and returns 0. I haven't realized that earlier, sorry. Michal

On Saturday, April 18, 2020 9:25:57 A.M. EDT Martin Kletzander wrote:

On Thu, Apr 16, 2020 at 11:57:45AM -0400, Mark Asselstine wrote:

...

It is possible and common to rename some devices, this is especially true for ethernet devices such as veth pairs.

In the udevEventHandleThread() we will be notified of this change but currently we only process "add", "change" and "remove" events. Renaming a device such as above results in a "move" event, not a "remove" followed by and "add" or vise versa. This change will add the new/destination device to our records but unfortunately there is no usable mechanism to identify the old/source device to remove it from the records. So this is admittedly only a partial fix.

There is, but it is only internal. We should ask for the function `udev_device_get_devpath_old()` to be publicized, if possible. That should give us the old name.

When I checked this using `udevadm monitor --property` I got two events for each rename, one kernel event (where the old path pointed to the previous name) and one udev event where the old path referred to the original name the device was created with.

For example when I created a virtual network device with the name "fdsa", then renamed it bunch of times, started udev monitoring and then renamed it from "first" to "second" this was the output of udevadm for that one particular rename:

monitor will print the received events for: UDEV - the event which udev sends out after rule processing KERNEL - the kernel uevent

KERNEL[36343.246068] move /devices/virtual/net/second (net) ACTION=move DEVPATH=/devices/virtual/net/second DEVPATH_OLD=/devices/virtual/net/first IFINDEX=3 INTERFACE=second SEQNUM=4609 SUBSYSTEM=net

UDEV [36343.246785] move /devices/virtual/net/fdsa (net) ACTION=move DEVPATH=/devices/virtual/net/fdsa DEVPATH_OLD=/devices/virtual/net/first IFINDEX=3 INTERFACE=fdsa SEQNUM=4609 SUBSYSTEM=net USEC_INITIALIZED=136864589

Thank you for a patch like this, I did not want to change the error just because "it's probably fine" until I can be sure that we do not lose the device (even though someone else is clearly managing it).

So I think the patches are great, and as much as I would like to add support for removing the old device if possible, probably using the function above, it will need some extra work. At least we should file a request for the udev function to be exposed.

Anyway, I am not against these patches and I would even be fine with changing the warning to info in 2/2, but I will wait for someone else to see this and comment (and I'd like to know your opinion on what I wrote as well). If that does not happen, I will retest them against (at that point) current master and push them.

Reviewed-by: Martin Kletzander <mkletzan@redhat.com>

Thanks Martin, I am in agreement with your analysis and assessment, and am happy that you are okay with the approach I took, I had my concerns. As you say, this is ideally something that needs to be taken up with the udev folks, as that is the proper place to start to correct this, they really do need to expose the src and dst in a move action. I tried several workarounds such as invalidating the device list and re-enumerating the devices again, and as you can see I opted to be blunt about this as there was no good workaround. Thanks for considering these patches. As always I will keep an eye out to ensure they don't create new problems. Mark

...

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> ---

src/node_device/node_device_udev.c | 5 +++++ 1 file changed, 5 insertions(+)

diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 8451903e8a..3149de8321 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1499,6 +1499,11 @@ udevHandleOneDevice(struct udev_device *device)

if (STREQ(action, "remove"))

return udevRemoveOneDevice(device);

+ if (STREQ(action, "move")) { + /* TODO: implement a way of finding and removing the old device */ + return udevAddOneDevice(device); + } +

return 0;

}