[libvirt] [PATCH v2] nodedev: Increase the netlink socket buffer size to the one used by udev
From: "ning.bo" <ning.bo9@zte.com.cn> When a number of SRIOV VFs (up to 128 on Intel XL710) is created: for i in `seq 0 1`; do echo 63 > /sys/class/net/<interface>/device/sriov_numvfs done libvirtd will then report "udev_monitor_receive_device returned NULL" error because the netlink socket buffer is not big enough (using GDB on libudev confirmed this with ENOBUFFS) and thus some udev events were dropped. This results in some devices being missing in the nodedev-list output. This patch overrides the system's rmem_max limit but for that, we need to make sure we've got root privileges. https://bugzilla.redhat.com/show_bug.cgi?id=1450960 Signed-off-by: ning.bo <ning.bo9@zte.com.cn> Signed-off-by: Erik Skultety <eskultet@redhat.com> --- Additionally, we might want to check for the errno, providing a specific message if such issue occurs again in a further non-specified point in time and return the generic, yet cryptic one for all other cases. src/node_device/node_device_udev.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 4ecb0b18f..0b3717397 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged, udev_monitor_enable_receiving(priv->udev_monitor); + /* mimic udevd's behaviour and override the systems rmem_max limit in case + * there's a significant number of device 'add' events + */ + if (geteuid() == 0) + udev_monitor_set_receive_buffer_size(priv->udev_monitor, + 128 * 1024 * 1024); + /* We register the monitor with the event callback so we are * notified by udev of device changes before we enumerate existing * devices because libvirt will simply recreate the device if we -- 2.13.0
On Fri, May 19, 2017 at 03:16:26PM +0200, Erik Skultety wrote:
From: "ning.bo" <ning.bo9@zte.com.cn>
Ping?
When a number of SRIOV VFs (up to 128 on Intel XL710) is created: for i in `seq 0 1`; do echo 63 > /sys/class/net/<interface>/device/sriov_numvfs done
libvirtd will then report "udev_monitor_receive_device returned NULL" error because the netlink socket buffer is not big enough (using GDB on libudev confirmed this with ENOBUFFS) and thus some udev events were dropped. This results in some devices being missing in the nodedev-list output. This patch overrides the system's rmem_max limit but for that, we need to make sure we've got root privileges.
https://bugzilla.redhat.com/show_bug.cgi?id=1450960
Signed-off-by: ning.bo <ning.bo9@zte.com.cn> Signed-off-by: Erik Skultety <eskultet@redhat.com> --- Additionally, we might want to check for the errno, providing a specific message if such issue occurs again in a further non-specified point in time and return the generic, yet cryptic one for all other cases.
src/node_device/node_device_udev.c | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 4ecb0b18f..0b3717397 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,
udev_monitor_enable_receiving(priv->udev_monitor);
+ /* mimic udevd's behaviour and override the systems rmem_max limit in case + * there's a significant number of device 'add' events + */ + if (geteuid() == 0) + udev_monitor_set_receive_buffer_size(priv->udev_monitor, + 128 * 1024 * 1024); + /* We register the monitor with the event callback so we are * notified by udev of device changes before we enumerate existing * devices because libvirt will simply recreate the device if we -- 2.13.0
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 05/19/2017 03:16 PM, Erik Skultety wrote:
From: "ning.bo" <ning.bo9@zte.com.cn>
When a number of SRIOV VFs (up to 128 on Intel XL710) is created: for i in `seq 0 1`; do echo 63 > /sys/class/net/<interface>/device/sriov_numvfs done
libvirtd will then report "udev_monitor_receive_device returned NULL" error because the netlink socket buffer is not big enough (using GDB on libudev confirmed this with ENOBUFFS) and thus some udev events were dropped. This results in some devices being missing in the nodedev-list output. This patch overrides the system's rmem_max limit but for that, we need to make sure we've got root privileges.
https://bugzilla.redhat.com/show_bug.cgi?id=1450960
Signed-off-by: ning.bo <ning.bo9@zte.com.cn> Signed-off-by: Erik Skultety <eskultet@redhat.com> --- Additionally, we might want to check for the errno, providing a specific message if such issue occurs again in a further non-specified point in time and return the generic, yet cryptic one for all other cases.
src/node_device/node_device_udev.c | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 4ecb0b18f..0b3717397 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,
udev_monitor_enable_receiving(priv->udev_monitor);
+ /* mimic udevd's behaviour and override the systems rmem_max limit in case + * there's a significant number of device 'add' events + */ + if (geteuid() == 0) + udev_monitor_set_receive_buffer_size(priv->udev_monitor, + 128 * 1024 * 1024); + /* We register the monitor with the event callback so we are * notified by udev of device changes before we enumerate existing * devices because libvirt will simply recreate the device if we
ACK and safe for freeze. Although on my system it works even without the check for euid. Michal
On Mon, May 29, 2017 at 02:16:17PM +0200, Michal Privoznik wrote:
On 05/19/2017 03:16 PM, Erik Skultety wrote:
From: "ning.bo" <ning.bo9@zte.com.cn>
When a number of SRIOV VFs (up to 128 on Intel XL710) is created: for i in `seq 0 1`; do echo 63 > /sys/class/net/<interface>/device/sriov_numvfs done
libvirtd will then report "udev_monitor_receive_device returned NULL" error because the netlink socket buffer is not big enough (using GDB on libudev confirmed this with ENOBUFFS) and thus some udev events were dropped. This results in some devices being missing in the nodedev-list output. This patch overrides the system's rmem_max limit but for that, we need to make sure we've got root privileges.
https://bugzilla.redhat.com/show_bug.cgi?id=1450960
Signed-off-by: ning.bo <ning.bo9@zte.com.cn> Signed-off-by: Erik Skultety <eskultet@redhat.com> --- Additionally, we might want to check for the errno, providing a specific message if such issue occurs again in a further non-specified point in time and return the generic, yet cryptic one for all other cases.
src/node_device/node_device_udev.c | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/src/node_device/node_device_udev.c b/src/node_device/node_device_udev.c index 4ecb0b18f..0b3717397 100644 --- a/src/node_device/node_device_udev.c +++ b/src/node_device/node_device_udev.c @@ -1728,6 +1728,13 @@ static int nodeStateInitialize(bool privileged,
udev_monitor_enable_receiving(priv->udev_monitor);
+ /* mimic udevd's behaviour and override the systems rmem_max limit in case + * there's a significant number of device 'add' events + */ + if (geteuid() == 0) + udev_monitor_set_receive_buffer_size(priv->udev_monitor, + 128 * 1024 * 1024); + /* We register the monitor with the event callback so we are * notified by udev of device changes before we enumerate existing * devices because libvirt will simply recreate the device if we
ACK and safe for freeze. Although on my system it works even without the check for euid.
Are you sure? Because the session daemon should not have the CAP_NET_ADMIN capability. Though it's true that since we don't care about the return value the check is unnecessary I give you that. Still, the check is harmless and we know that it will fail for non-privileged user, so I'm going to keep it here. Thanks for review, Erik
Michal
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Fri, May 19, 2017 at 03:16:26PM +0200, Erik Skultety wrote:
From: "ning.bo" <ning.bo9@zte.com.cn>
When a number of SRIOV VFs (up to 128 on Intel XL710) is created: for i in `seq 0 1`; do echo 63 > /sys/class/net/<interface>/device/sriov_numvfs done
libvirtd will then report "udev_monitor_receive_device returned NULL" error because the netlink socket buffer is not big enough (using GDB on libudev confirmed this with ENOBUFFS) and thus some udev events were dropped. This results in some devices being missing in the nodedev-list output. This patch overrides the system's rmem_max limit but for that, we need to make sure we've got root privileges.
https://bugzilla.redhat.com/show_bug.cgi?id=1450960
Signed-off-by: ning.bo <ning.bo9@zte.com.cn> Signed-off-by: Erik Skultety <eskultet@redhat.com> --- Additionally, we might want to check for the errno, providing a specific message if such issue occurs again in a further non-specified point in time and return the generic, yet cryptic one for all other cases.
src/node_device/node_device_udev.c | 7 +++++++ 1 file changed, 7 insertions(+)
This has broken the build on older systems as udev doesn't have this method. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Daniel P. Berrange -
Erik Skultety -
Michal Privoznik