[libvirt] NL_RECV RETURNED WITH ERROR: NO BUFFER SPACE AVAILABLE ERROR FOR IXGBE LOAD

Hi experts, Release: OL7 Kernel: 3.10.0-121.el7.x86_64 Noticed below error on OL7 server, while loading Intel 10gigabit nic driver module , ixgbe in syslog -------------------------------------------------------------------- journal: nl_recv returned with error: No buffer space available ------------------------------------------------------------------- Complete syslog content for ixgbe module load : un 16 20:46:10 ca-ostest432 kernel: ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k Jun 16 20:46:10 ca-ostest432 kernel: ixgbe: Copyright (c) 1999-2013 Intel Corporation. Jun 16 20:46:10 ca-ostest432 kvm: 1 guest now active Jun 16 20:46:10 ca-ostest432 kvm: 0 guests now active Jun 16 20:46:10 ca-ostest432 kernel: ixgbe 0000:13:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 Jun 16 20:46:10 ca-ostest432 kernel: ixgbe 0000:13:00.0: (PCI Express:5.0GT/s:Width x8) 00:1b:21:c8:24:74 Jun 16 20:46:10 ca-ostest432 kernel: ixgbe 0000:13:00.0: MAC: 2, PHY: 9, SFP+: 3, PBA No: E70856-007 Jun 16 20:46:10 ca-ostest432 kernel: ixgbe 0000:13:00.0: PCI Express bandwidth of 32GT/s available Jun 16 20:46:10 ca-ostest432 kernel: ixgbe 0000:13:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) Jun 16 20:46:10 ca-ostest432 journal: nl_recv returned with error: No buffer space available Jun 16 20:46:11 ca-ostest432 journal: nl_recv returned with error: No buffer space available Jun 16 20:46:11 ca-ostest432 systemd-udevd[4939]: renamed network interface eth0 to ens2f0 Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.0: Intel(R) 10 Gigabit Network Connection Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.1: (PCI Express:5.0GT/s:Width x8) 00:1b:21:c8:24:75 Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.1: MAC: 2, PHY: 9, SFP+: 4, PBA No: E70856-007 Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.1: PCI Express bandwidth of 32GT/s available Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) Jun 16 20:46:11 ca-ostest432 systemd-udevd[4984]: renamed network interface eth0 to ens2f1 Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.1: Intel(R) 10 Gigabit Network Connection Jun 16 20:46:11 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): carrier is OFF (but ignored) Jun 16 20:46:11 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): new Ethernet device (driver: 'ixgbe' ifindex: 14) Jun 16 20:46:11 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): exported as /org/freedesktop/NetworkManager/Devices/13 Jun 16 20:46:11 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2] Jun 16 20:46:11 ca-ostest432 journal: nl_recv returned with error: No buffer space available Jun 16 20:46:11 ca-ostest432 kernel: ixgbe 0000:13:00.0: registered PHC device on ens2f0 Jun 16 20:46:12 ca-ostest432 kernel: IPv6: ADDRCONF(NETDEV_UP): ens2f0: link is not ready Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): preparing device Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): carrier is OFF (but ignored) Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): new Ethernet device (driver: 'ixgbe' ifindex: 15) Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): exported as /org/freedesktop/NetworkManager/Devices/14 Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2] Jun 16 20:46:12 ca-ostest432 journal: nl_recv returned with error: No buffer space available Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): preparing device Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): device state change: unavailable -> disconnected (reason 'none') [20 30 0] Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): device state change: unavailable -> disconnected (reason 'none') [20 30 0] Jun 16 20:46:12 ca-ostest432 kernel: ixgbe 0000:13:00.0 ens2f0: detected SFP+: 3 Jun 16 20:46:12 ca-ostest432 kernel: ixgbe 0000:13:00.1: registered PHC device on ens2f1 Jun 16 20:46:12 ca-ostest432 kernel: IPv6: ADDRCONF(NETDEV_UP): ens2f1: link is not ready Jun 16 20:46:12 ca-ostest432 kernel: ixgbe 0000:13:00.1 ens2f1: detected SFP+: 4 Jun 16 20:46:12 ca-ostest432 kernel: ixgbe 0000:13:00.1 ens2f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX Jun 16 20:46:12 ca-ostest432 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens2f1: link becomes ready Jun 16 20:46:12 ca-ostest432 NetworkManager[1316]: <info> (ens2f1): link connected Jun 16 20:46:12 ca-ostest432 kernel: ixgbe 0000:13:00.0 ens2f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX Jun 16 20:46:12 ca-ostest432 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens2f0: link becomes ready Jun 16 20:46:13 ca-ostest432 NetworkManager[1316]: <info> (ens2f0): link connected --------------------------------------------------------------------------------------------- Issue is not seen for other nic drivers when loaded , i.e verified with be2net ,e1000e , igb drivers , so it looks like specific to ixgbe driver module. This problem can be addressed by increasing the size of the socket receive buffer via nl_socket_set_buffer_size(). However, since reliable transmissions from kernel to user are impossible in any case, I don't know if we need to ensure enough buffer and how much it will be enough. In my test, 32768*4 bytes of receive buffer size is ok. Below is my patch. From: rpm-build <rpm-build> Date: Mon, 16 Jun 2014 20:05:08 +0800 Subject: [PATCH] set nl socket buffer size to 32768*4 Signed-off-by: Vaughan Cao <vaughan.cao(a)oracle.com&gt; --- src/util/virnetlink.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/util/virnetlink.c b/src/util/virnetlink.c index 5849ed7..cf44ad2 100644 --- a/src/util/virnetlink.c +++ b/src/util/virnetlink.c @@ -561,6 +561,15 @@ virNetlinkEventServiceStart(unsigned int protocol, unsigned int groups) goto error_server; } + if (nl_socket_set_buffer_size(srv->netlinknh, (32768*4), 0)) { + virReportSystemError(errno, "%s", + _("cannot set netlink socket buffer size to 32768*4")); + goto error_server; + } else { + virReportSystemError(0, "%s", + _("set netlink socket buffer size to 32768*4")); + } + if ((srv->eventwatch = virEventAddHandle(fd, VIR_EVENT_HANDLE_READABLE, virNetlinkEventCallback, -- 1.9.0 -- Regards, Vaughan