[libvirt-users] multicast traffic no longer working after reboot
Hi, today a couple of Hypervisors got restarted due to a power outage. While the systems and the VMs on the are running fine multicast traffic between the guests is no longer working. I generate multicast udp packet using iperf in the guest and check on the host using tcpdump: tcpdump vnet0 => multicast packets are visible tcpdump vlanbr8 => multicast packets are visible tcpdump em1.8 => no multicast packets visible vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the physical interface for vlan 8. I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_* with a system where multicast works fine and see no differences. brctl showstp also shows "mc router" to be 1 for both interfaces and the bridge. All settings in /proc/sys/net/bridge/* are 0. Any ideas why the bridge doesn't seem to forward multicast packets? iperf with unicast udp also works fine so it's just multicast that is the problem. Regards, Dennis
On 31.07.2014 15:14, Dennis Jacobfeuerborn wrote:
Hi, today a couple of Hypervisors got restarted due to a power outage. While the systems and the VMs on the are running fine multicast traffic between the guests is no longer working.
I generate multicast udp packet using iperf in the guest and check on the host using tcpdump:
tcpdump vnet0 => multicast packets are visible tcpdump vlanbr8 => multicast packets are visible tcpdump em1.8 => no multicast packets visible
vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the physical interface for vlan 8.
I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_* with a system where multicast works fine and see no differences. brctl showstp also shows "mc router" to be 1 for both interfaces and the bridge. All settings in /proc/sys/net/bridge/* are 0.
Any ideas why the bridge doesn't seem to forward multicast packets? iperf with unicast udp also works fine so it's just multicast that is the problem.
I did some further testing and strangely enough after booting the HV and the guest things work as expected but after 2-3 minutes I suddenly don't see the the packets anymore on em1.8. This is really bizarre behavior. Regards, Dennis
On 31.07.2014 17:23, Dennis Jacobfeuerborn wrote:
On 31.07.2014 15:14, Dennis Jacobfeuerborn wrote:
Hi, today a couple of Hypervisors got restarted due to a power outage. While the systems and the VMs on the are running fine multicast traffic between the guests is no longer working.
I generate multicast udp packet using iperf in the guest and check on the host using tcpdump:
tcpdump vnet0 => multicast packets are visible tcpdump vlanbr8 => multicast packets are visible tcpdump em1.8 => no multicast packets visible
vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the physical interface for vlan 8.
I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_* with a system where multicast works fine and see no differences. brctl showstp also shows "mc router" to be 1 for both interfaces and the bridge. All settings in /proc/sys/net/bridge/* are 0.
Any ideas why the bridge doesn't seem to forward multicast packets? iperf with unicast udp also works fine so it's just multicast that is the problem.
I did some further testing and strangely enough after booting the HV and the guest things work as expected but after 2-3 minutes I suddenly don't see the the packets anymore on em1.8. This is really bizarre behavior.
So after a lot of digging I seem to have found the cause of the issue but I'm not sure what the proper fix is. The cause seems to be a kernel "regression": https://bugzilla.redhat.com/show_bug.cgi?id=880035#c56 It seems that multicast now requires explicitly setting /sys/class/<bridge>/bridge/multicast_querier to 1. What is puzzling about this is that im wonder why this behavior changeing patch was included in the RHEL/CentOS kernel. This is the first time I ever ran into this kind of problem and previously multicast seemed to work just out of the box. The other issue is that I'm not sure how to make this change permanent. When changing somthing under /proc/sys I can put the value in /etc/sysctl.conf but no such config file exists for /sys/. There is a snippet in the bug report but that only sets the flag for specific bridges. What I'm really looking for is a way to get back to the old behaviour where I can start a bridge and have multicast work by default. Regards, Dennis
This sounds a lot like you have IGMP snooping enabled, but you don't have anything sending the periodic IGMP queries. We've seen things like this when the network is configured for IGMP, but no querier exists. Things will work correctly for a few minutes after they start up, then suddenly stop after the initial IGMP join message times out. On 7/31/2014 1:44 PM, Dennis Jacobfeuerborn wrote:
On 31.07.2014 17:23, Dennis Jacobfeuerborn wrote:
Hi, today a couple of Hypervisors got restarted due to a power outage. While the systems and the VMs on the are running fine multicast traffic between the guests is no longer working.
I generate multicast udp packet using iperf in the guest and check on the host using tcpdump:
tcpdump vnet0 => multicast packets are visible tcpdump vlanbr8 => multicast packets are visible tcpdump em1.8 => no multicast packets visible
vlanbr8 is the bridge, vnet0 the interface of the guest and em1.8 the physical interface for vlan 8.
I compared the settings in /sys/class/net/vlanbr8/bridge/multicast_* with a system where multicast works fine and see no differences. brctl showstp also shows "mc router" to be 1 for both interfaces and the bridge. All settings in /proc/sys/net/bridge/* are 0.
Any ideas why the bridge doesn't seem to forward multicast packets? iperf with unicast udp also works fine so it's just multicast that is the problem. I did some further testing and strangely enough after booting the HV and
On 31.07.2014 15:14, Dennis Jacobfeuerborn wrote: the guest things work as expected but after 2-3 minutes I suddenly don't see the the packets anymore on em1.8. This is really bizarre behavior. So after a lot of digging I seem to have found the cause of the issue but I'm not sure what the proper fix is.
The cause seems to be a kernel "regression": https://bugzilla.redhat.com/show_bug.cgi?id=880035#c56
It seems that multicast now requires explicitly setting /sys/class/<bridge>/bridge/multicast_querier to 1.
What is puzzling about this is that im wonder why this behavior changeing patch was included in the RHEL/CentOS kernel. This is the first time I ever ran into this kind of problem and previously multicast seemed to work just out of the box.
The other issue is that I'm not sure how to make this change permanent. When changing somthing under /proc/sys I can put the value in /etc/sysctl.conf but no such config file exists for /sys/.
There is a snippet in the bug report but that only sets the flag for specific bridges. What I'm really looking for is a way to get back to the old behaviour where I can start a bridge and have multicast work by default.
Regards, Dennis
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
participants (2)
-
Brian Rak -
Dennis Jacobfeuerborn