[libvirt] LXC: autostart feature does set all interfaces to state up.
Hi! If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side. Most of the time only the first veth device is in state up, all others are down. Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers. This is not userns related, just retested with libvirt of today. Thanks, //richard
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed... maybe you should wait some seconds for the starting of these containers. Thanks!
Hi, Am 03.07.2013 12:04, schrieb Gao feng:
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up. On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up. Thanks, //richard
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script. [root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf Domain test-a marked as autostarted Domain test-b defined from container_b.conf Domain test-b marked as autostarted Domain test-c defined from container_c.conf Domain test-c marked as autostarted Domain test-d defined from container_d.conf Domain test-d marked as autostarted Domain test-e defined from container_e.conf Domain test-e marked as autostarted Domain test-f defined from container_f.conf Domain test-f marked as autostarted 2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 Can you post your libvirt debug log? Thanks
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file. 43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 100 Thanks, //richard
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 100
strange, I can not see veth related error message from your log. seems like all of the veth devices of host had been up but for some reasons they become down. can you also show me the bridge information on your host? brctl show virbr0 and your default net configuration virsh -c lxc:/// net-dumpxml default and your dmesg Thanks!
Am 08.07.2013 05:54, schrieb Gao feng:
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy. 1. Define a few containers (5 in my case) 2. Run "virsh autostart ..." on each one. 3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on the host side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 100
strange, I can not see veth related error message from your log.
seems like all of the veth devices of host had been up but for some reasons they become down.
I think libvirt has to do "ip link set dev vethX up". Otherwise the device state is undefined.
can you also show me the bridge information on your host? brctl show virbr0
bridge name bridge id STP enabled interfaces virbr0 8000.2e63ddca07fa yes veth0 veth1 veth2 veth3 veth4 veth5
and your default net configuration virsh -c lxc:/// net-dumpxml default
<network> <name>default</name> <uuid>a81b9f40-d279-4e96-8cec-6a906323f162</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0' /> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254' /> </dhcp> </ip> </network> Thanks, //richard
On 07/09/2013 09:11 PM, Richard Weinberger wrote:
Am 08.07.2013 05:54, schrieb Gao feng:
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
Hi, On 07/01/2013 03:45 PM, Richard Weinberger wrote: > Hi! > > If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to > up some veth interfaces on the host side. > > Most of the time only the first veth device is in state up, all others are down. > > Reproducing is easy. > 1. Define a few containers (5 in my case) > 2. Run "virsh autostart ..." on each one. > 3. stop/start libvirtd > > You'll observe that all containers are running, but "ip a" will report on the host > side that not all veth devices are up and are not usable within the containers. > > This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 100
strange, I can not see veth related error message from your log.
seems like all of the veth devices of host had been up but for some reasons they become down.
I think libvirt has to do "ip link set dev vethX up". Otherwise the device state is undefined.
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down. I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information. Thanks
Am 10.07.2013 03:20, schrieb Gao feng:
On 07/09/2013 09:11 PM, Richard Weinberger wrote:
Am 08.07.2013 05:54, schrieb Gao feng:
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng: > Hi, > On 07/01/2013 03:45 PM, Richard Weinberger wrote: >> Hi! >> >> If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to >> up some veth interfaces on the host side. >> >> Most of the time only the first veth device is in state up, all others are down. >> >> Reproducing is easy. >> 1. Define a few containers (5 in my case) >> 2. Run "virsh autostart ..." on each one. >> 3. stop/start libvirtd >> >> You'll observe that all containers are running, but "ip a" will report on the host >> side that not all veth devices are up and are not usable within the containers. >> >> This is not userns related, just retested with libvirt of today. > > I can not reproduce this problem on my test bed...
Strange.
> maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 100
strange, I can not see veth related error message from your log.
seems like all of the veth devices of host had been up but for some reasons they become down.
I think libvirt has to do "ip link set dev vethX up". Otherwise the device state is undefined.
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down. ---cut--- [ 4.167119] systemd[1]: Starting File System Check on Root Device... [ 5.150740] mount (725) used greatest stack depth: 4616 bytes left [ 5.397652] systemd-udevd[755]: starting version 195 [ 6.155648] EXT4-fs (vda2): re-mounted. Opts: acl,user_xattr [ 6.369755] systemd-journald[730]: Received SIGUSR1 [ 6.877588] Adding 690172k swap on /dev/vda1. Priority:-1 extents:1 across:690172k [ 8.205295] ip (1446) used greatest stack depth: 4584 bytes left [ 9.397258] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 9.397275] 8021q: adding VLAN 0 to HW filter on device eth0 [ 9.397733] ip (1690) used greatest stack depth: 4456 bytes left [ 11.399834] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 11.401711] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 41.179748] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready [ 42.126303] cgroup: libvirtd (2736) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 42.126307] cgroup: "memory" requires setting use_hierarchy to 1 on the root. [ 42.126599] cgroup: libvirtd (2736) created nested cgroup for controller "blkio" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 42.145511] ip (2843) used greatest stack depth: 4136 bytes left [ 42.149231] device veth0 entered promiscuous mode [ 42.152167] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready [ 42.559264] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready [ 42.559336] virbr0: topology change detected, propagating [ 42.559343] virbr0: port 1(veth0) entered forwarding state [ 42.559359] virbr0: port 1(veth0) entered forwarding state [ 42.559398] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready [ 42.702130] ip (3070) used greatest stack depth: 4088 bytes left [ 42.708164] device veth1 entered promiscuous mode [ 42.712190] IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready [ 42.712197] virbr0: topology change detected, propagating [ 42.712202] virbr0: port 2(veth1) entered forwarding state [ 42.712212] virbr0: port 2(veth1) entered forwarding state [ 43.147155] virbr0: port 2(veth1) entered disabled state [ 43.225250] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready [ 43.225322] virbr0: topology change detected, propagating [ 43.225328] virbr0: port 2(veth1) entered forwarding state [ 43.225358] virbr0: port 2(veth1) entered forwarding state [ 43.325556] ip (3293) used greatest stack depth: 4064 bytes left [ 43.330133] device veth2 entered promiscuous mode [ 43.334201] IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready [ 43.334208] virbr0: topology change detected, propagating [ 43.334214] virbr0: port 3(veth2) entered forwarding state [ 43.334224] virbr0: port 3(veth2) entered forwarding state [ 43.613184] IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready [ 43.767050] ip (3521) used greatest stack depth: 3992 bytes left [ 43.773832] device veth3 entered promiscuous mode [ 43.778169] IPv6: ADDRCONF(NETDEV_UP): veth3: link is not ready [ 43.778177] virbr0: topology change detected, propagating [ 43.778182] virbr0: port 4(veth3) entered forwarding state [ 43.778193] virbr0: port 4(veth3) entered forwarding state [ 44.076299] IPv6: ADDRCONF(NETDEV_CHANGE): veth3: link becomes ready [ 44.153214] device veth4 entered promiscuous mode [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state [ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state [ 45.348548] virbr0: port 6(veth5) entered disabled state [ 45.348837] ip (4349) used greatest stack depth: 3944 bytes left ---cut--- Thanks, //richard
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled. I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled. So, can you run kernel with the below debug patch? diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p); diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head) list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); } @@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } } if (dev->flags & IFF_UP && Thanks!
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); }
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } }
if (dev->flags & IFF_UP &&
Thanks!
There you go: [ 2.517243] systemd-journald[729]: Received SIGUSR1 [ 3.427222] Adding 690172k swap on /dev/vda1. Priority:-1 extents:1 across:690172k [ 5.703781] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 5.703796] 8021q: adding VLAN 0 to HW filter on device eth0 [ 7.708579] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 7.709232] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 126.002595] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready [ 126.450081] cgroup: libvirtd (2763) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 126.450085] cgroup: "memory" requires setting use_hierarchy to 1 on the root. [ 126.450365] cgroup: libvirtd (2763) created nested cgroup for controller "blkio" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 126.463191] device veth0 entered promiscuous mode [ 126.468207] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready [ 126.468216] virbr0: topology change detected, propagating [ 126.468222] virbr0: port 1(veth0) entered forwarding state [ 126.468237] virbr0: port 1(veth0) entered forwarding state [ 126.468265] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready [ 126.689269] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready [ 126.786773] ip (3094) used greatest stack depth: 4072 bytes left [ 126.793169] device veth1 entered promiscuous mode [ 126.799334] IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready [ 126.799342] virbr0: topology change detected, propagating [ 126.799348] virbr0: port 2(veth1) entered forwarding state [ 126.799360] virbr0: port 2(veth1) entered forwarding state [ 126.903181] CPU: 0 PID: 769 Comm: kworker/0:2 Not tainted 3.10.0+ #20 [ 126.903187] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 126.903226] Workqueue: events linkwatch_event [ 126.903231] ffff88003bee2800 ffff88003d6b1c38 ffffffff8172b2a6 ffff88003d6b1c78 [ 126.903235] ffffffff81666d8e 0000000100000000 ffff88003bc11800 ffff88003bee2800 [ 126.903240] ffff88003cf5d000 ffff88003cf5d000 0000000000000000 ffff88003d6b1ca8 [ 126.903244] Call Trace: [ 126.903268] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 126.903288] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100 [ 126.903294] [<ffffffff81663e38>] br_port_carrier_check+0xd8/0xe0 [ 126.903299] [<ffffffff816655d8>] br_device_event+0x168/0x200 [ 126.903326] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70 [ 126.903332] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10 [ 126.903337] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20 [ 126.903347] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60 [ 126.903352] [<ffffffff815484b2>] netdev_state_change+0x22/0x40 [ 126.903357] [<ffffffff8155bd47>] linkwatch_do_dev+0x37/0x60 [ 126.903362] [<ffffffff8155bf56>] __linkwatch_run_queue+0xe6/0x170 [ 126.903367] [<ffffffff8155c000>] linkwatch_event+0x20/0x30 [ 126.903373] [<ffffffff810567bf>] process_one_work+0x17f/0x420 [ 126.903377] [<ffffffff81056ea9>] worker_thread+0x119/0x370 [ 126.903382] [<ffffffff81056d90>] ? rescuer_thread+0x2f0/0x2f0 [ 126.903388] [<ffffffff8105d42b>] kthread+0xbb/0xc0 [ 126.903394] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50 [ 126.903404] [<ffffffff8173326c>] ret_from_fork+0x7c/0xb0 [ 126.903410] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50 [ 126.903414] virbr0: port 2(veth1) entered disabled state [ 127.038275] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready [ 127.038360] virbr0: topology change detected, propagating [ 127.038368] virbr0: port 2(veth1) entered forwarding state [ 127.038382] virbr0: port 2(veth1) entered forwarding state [ 127.124475] ip (3296) used greatest stack depth: 4016 bytes left [ 127.129185] device veth2 entered promiscuous mode [ 127.134253] IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready [ 127.134261] virbr0: topology change detected, propagating [ 127.134267] virbr0: port 3(veth2) entered forwarding state [ 127.134278] virbr0: port 3(veth2) entered forwarding state [ 127.325301] IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready [ 127.441199] device veth3 entered promiscuous mode [ 127.445205] IPv6: ADDRCONF(NETDEV_UP): veth3: link is not ready [ 127.445213] virbr0: topology change detected, propagating [ 127.445218] virbr0: port 4(veth3) entered forwarding state [ 127.445228] virbr0: port 4(veth3) entered forwarding state [ 127.703335] IPv6: ADDRCONF(NETDEV_CHANGE): veth3: link becomes ready [ 127.814220] device veth4 entered promiscuous mode [ 127.818206] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 127.818215] virbr0: topology change detected, propagating [ 127.818221] virbr0: port 5(veth4) entered forwarding state [ 127.818232] virbr0: port 5(veth4) entered forwarding state [ 127.903178] CPU: 1 PID: 352 Comm: kworker/1:1 Not tainted 3.10.0+ #20 [ 127.903183] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 127.903191] Workqueue: events linkwatch_event [ 127.903194] ffff88003bee2800 ffff88003db27c38 ffffffff8172b2a6 ffff88003db27c78 [ 127.903198] ffffffff81666d8e 0000000100000000 ffff8800374b0400 ffff88003bee2800 [ 127.903201] ffff88003b39e000 ffff88003b39e000 0000000000000000 ffff88003db27ca8 [ 127.903204] Call Trace: [ 127.903211] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 127.903216] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100 [ 127.903221] [<ffffffff81663e38>] br_port_carrier_check+0xd8/0xe0 [ 127.903224] [<ffffffff816655d8>] br_device_event+0x168/0x200 [ 127.903230] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70 [ 127.903234] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10 [ 127.903238] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20 [ 127.903242] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60 [ 127.903245] [<ffffffff815484b2>] netdev_state_change+0x22/0x40 [ 127.903249] [<ffffffff8155bd47>] linkwatch_do_dev+0x37/0x60 [ 127.903253] [<ffffffff8155bf56>] __linkwatch_run_queue+0xe6/0x170 [ 127.903256] [<ffffffff8155c000>] linkwatch_event+0x20/0x30 [ 127.903260] [<ffffffff810567bf>] process_one_work+0x17f/0x420 [ 127.903263] [<ffffffff81056ea9>] worker_thread+0x119/0x370 [ 127.903266] [<ffffffff81056d90>] ? rescuer_thread+0x2f0/0x2f0 [ 127.903270] [<ffffffff8105d42b>] kthread+0xbb/0xc0 [ 127.903274] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50 [ 127.903278] [<ffffffff8173326c>] ret_from_fork+0x7c/0xb0 [ 127.903283] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50 [ 127.903287] virbr0: port 5(veth4) entered disabled state [ 128.089226] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 128.089281] virbr0: topology change detected, propagating [ 128.089286] virbr0: port 5(veth4) entered forwarding state [ 128.089297] virbr0: port 5(veth4) entered forwarding state [ 128.098859] CPU: 1 PID: 3963 Comm: ip Not tainted 3.10.0+ #20 [ 128.098864] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 128.098867] ffff88003b21c000 ffff8800374cd798 ffffffff8172b2a6 ffff8800374cd7b8 [ 128.098871] ffffffff8154be04 ffff88003b21c000 0000000000001103 ffff8800374cd7e8 [ 128.098874] ffffffff8154be60 0000000000000010 ffff88003b21c000 ffff8800374cd918 [ 128.098877] Call Trace: [ 128.098885] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 128.098890] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90 [ 128.098894] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 128.098898] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 128.098921] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 128.098937] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 128.098941] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 128.098944] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 128.098948] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 128.098964] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 128.098971] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 128.098974] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 128.098983] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 128.098987] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 128.098990] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 128.098994] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 128.099000] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 128.099015] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 128.099067] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 128.099072] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 128.099080] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 128.099091] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 128.099095] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 128.099098] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 128.099101] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 128.099108] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 128.099259] CPU: 1 PID: 3963 Comm: ip Not tainted 3.10.0+ #20 [ 128.099262] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 128.099263] ffff88003bee2800 ffff8800374cd698 ffffffff8172b2a6 ffff8800374cd6d8 [ 128.099267] ffffffff81666d8e ffff88003ccd1008 0000000000000002 ffff88003b10f400 [ 128.099270] ffff88003bee2800 ffff88003b21c000 0000000000000000 ffff8800374cd718 [ 128.099273] Call Trace: [ 128.099277] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 128.099282] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100 [ 128.099285] [<ffffffff81665668>] br_device_event+0x1f8/0x200 [ 128.099289] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400 [ 128.099295] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70 [ 128.099299] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10 [ 128.099302] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20 [ 128.099306] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60 [ 128.099310] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90 [ 128.099313] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 128.099317] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 128.099320] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 128.099324] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 128.099328] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 128.099331] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 128.099335] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 128.099339] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 128.099343] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 128.099346] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 128.099350] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 128.099353] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 128.099357] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 128.099360] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 128.099364] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 128.099368] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 128.099371] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 128.099376] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 128.099380] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 128.099384] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 128.099387] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 128.099390] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 128.099393] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 128.099396] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 128.099399] virbr0: port 3(veth2) entered disabled state [ 128.103318] ip (3963) used greatest stack depth: 3864 bytes left [ 128.203201] device veth5 entered promiscuous mode [ 128.207856] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 128.207865] virbr0: topology change detected, propagating [ 128.207872] virbr0: port 6(veth5) entered forwarding state [ 128.207885] virbr0: port 6(veth5) entered forwarding state [ 128.454311] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 128.564812] CPU: 1 PID: 4299 Comm: ip Not tainted 3.10.0+ #20 [ 128.564817] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 128.564819] ffff88003b21e000 ffff880037531798 ffffffff8172b2a6 ffff8800375317b8 [ 128.564823] ffffffff8154be04 ffff88003b21e000 0000000000001103 ffff8800375317e8 [ 128.564826] ffffffff8154be60 0000000000000010 ffff88003b21e000 ffff880037531918 [ 128.564830] Call Trace: [ 128.564838] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 128.564843] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90 [ 128.564846] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 128.564851] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 128.564856] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 128.564860] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 128.564864] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 128.564878] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 128.564882] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 128.564888] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 128.564893] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 128.564896] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 128.564901] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 128.564904] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 128.564907] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 128.564911] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 128.564916] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 128.564920] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 128.564925] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 128.564929] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 128.564934] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 128.564939] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 128.564942] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 128.564945] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 128.564948] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 128.564953] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 128.565135] CPU: 1 PID: 4299 Comm: ip Not tainted 3.10.0+ #20 [ 128.565137] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 128.565138] ffff88003bee2800 ffff880037531698 ffffffff8172b2a6 ffff8800375316d8 [ 128.565142] ffffffff81666d8e ffff88003b17e808 0000000000000002 ffff88003b394400 [ 128.565145] ffff88003bee2800 ffff88003b21e000 0000000000000000 ffff880037531718 [ 128.565148] Call Trace: [ 128.565152] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 128.565156] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100 [ 128.565162] [<ffffffff81665668>] br_device_event+0x1f8/0x200 [ 128.565166] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400 [ 128.565170] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70 [ 128.565174] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10 [ 128.565177] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20 [ 128.565181] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60 [ 128.565185] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90 [ 128.565190] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 128.565193] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 128.565197] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 128.565201] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 128.565205] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 128.565209] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 128.565213] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 128.565218] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 128.565221] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 128.565225] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 128.565229] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 128.565232] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 128.565235] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 128.565239] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 128.565243] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 128.565246] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 128.565250] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 128.565254] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 128.565258] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 128.565263] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 128.565266] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 128.565269] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 128.565272] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 128.565275] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 128.565280] virbr0: port 4(veth3) entered disabled state [ 128.796890] CPU: 1 PID: 4432 Comm: ip Not tainted 3.10.0+ #20 [ 128.796896] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 128.796899] ffff88003b39e000 ffff88003779f798 ffffffff8172b2a6 ffff88003779f7b8 [ 128.796905] ffffffff8154be04 ffff88003b39e000 0000000000001103 ffff88003779f7e8 [ 128.796920] ffffffff8154be60 0000000000000010 ffff88003b39e000 ffff88003779f918 [ 128.796924] Call Trace: [ 128.796931] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 128.796936] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90 [ 128.796940] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 128.796945] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 128.796950] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 128.796955] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 128.796959] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 128.796962] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 128.796966] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 128.796971] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 128.796976] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 128.796979] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 128.796983] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 128.796986] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 128.796990] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 128.796993] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 128.796998] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 128.797003] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 128.797013] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 128.797049] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 128.797054] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 128.797059] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 128.797063] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 128.797066] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 128.797069] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 128.797073] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 128.797182] CPU: 1 PID: 4432 Comm: ip Not tainted 3.10.0+ #20 [ 128.797184] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 128.797186] ffff88003bee2800 ffff88003779f698 ffffffff8172b2a6 ffff88003779f6d8 [ 128.797189] ffffffff81666d8e ffff88003b281808 0000000000000002 ffff8800374b0400 [ 128.797192] ffff88003bee2800 ffff88003b39e000 0000000000000000 ffff88003779f718 [ 128.797195] Call Trace: [ 128.797199] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 128.797203] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100 [ 128.797207] [<ffffffff81665668>] br_device_event+0x1f8/0x200 [ 128.797211] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400 [ 128.797215] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70 [ 128.797219] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10 [ 128.797223] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20 [ 128.797227] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60 [ 128.797230] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90 [ 128.797234] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 128.797237] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 128.797241] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 128.797245] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 128.797248] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 128.797251] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 128.797255] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 128.797260] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 128.797263] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 128.797266] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 128.797270] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 128.797273] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 128.797277] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 128.797280] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 128.797284] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 128.797288] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 128.797291] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 128.797301] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 128.797306] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 128.797310] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 128.797313] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 128.797316] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 128.797319] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 128.797322] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 128.797325] virbr0: port 5(veth4) entered disabled state [ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8 [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8 [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918 [ 129.084426] Call Trace: [ 129.084433] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 129.084438] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90 [ 129.084442] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 129.084446] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 129.084451] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 129.084456] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 129.084460] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 129.084463] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 129.084467] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 129.084473] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 129.084477] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 129.084481] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 129.084485] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 129.084488] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 129.084492] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 129.084495] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 129.084501] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 129.084505] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 129.084509] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 129.084513] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 129.084518] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 129.084523] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 129.084526] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 129.084530] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 129.084533] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 129.084537] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 129.084672] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084674] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084676] ffff88003bee2800 ffff88003ce7f698 ffffffff8172b2a6 ffff88003ce7f6d8 [ 129.084679] ffffffff81666d8e ffff88003764c008 0000000000000002 ffff8800374b1400 [ 129.084682] ffff88003bee2800 ffff88003760d000 0000000000000000 ffff88003ce7f718 [ 129.084685] Call Trace: [ 129.084689] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b [ 129.084694] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100 [ 129.084697] [<ffffffff81665668>] br_device_event+0x1f8/0x200 [ 129.084701] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400 [ 129.084706] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70 [ 129.084710] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10 [ 129.084713] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20 [ 129.084717] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60 [ 129.084721] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90 [ 129.084724] [<ffffffff8154be60>] dev_change_flags+0x40/0x70 [ 129.084728] [<ffffffff81559a31>] do_setlink+0x351/0x980 [ 129.084731] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330 [ 129.084735] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0 [ 129.084739] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0 [ 129.084742] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60 [ 129.084746] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240 [ 129.084761] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250 [ 129.084765] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290 [ 129.084768] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20 [ 129.084772] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0 [ 129.084776] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30 [ 129.084779] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0 [ 129.084782] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370 [ 129.084787] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0 [ 129.084790] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0 [ 129.084794] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40 [ 129.084798] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360 [ 129.084802] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200 [ 129.084806] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0 [ 129.084809] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320 [ 129.084812] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80 [ 129.084815] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20 [ 129.084818] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b [ 129.084821] virbr0: port 6(veth5) entered disabled state Thanks, //richard
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); }
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } }
if (dev->flags & IFF_UP &&
Thanks!
There you go:
Thank you very much.
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8 [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8 [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918 [ 129.084426] Call Trace:
[ 129.084821] virbr0: port 6(veth5) entered disabled state
I can confirm it's the ip command disable the veth device now. but I still don't know who calls ip and why. I search the libvirt code, there are no codes calling "ip link set xxx down". It's so strange... Give you an advice, modify the code of ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git). use read /proc/<getppid>/comm to trace which command calls ip. ip/iplink.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/ip/iplink.c b/ip/iplink.c index dc98019..ec7ad87 100644 --- a/ip/iplink.c +++ b/ip/iplink.c @@ -282,6 +282,7 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req, int mtu = -1; int netns = -1; int vf = -1; + int down = 0; int numtxqueues = -1; int numrxqueues = -1; @@ -293,8 +294,25 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req, req->i.ifi_change |= IFF_UP; req->i.ifi_flags |= IFF_UP; } else if (strcmp(*argv, "down") == 0) { + pid_t ppid = getppid(); + char file[30]; + char comm[20]; + int fd; + memset(file, 0, sizeof(file)); + memset(comm, 0, sizeof(comm)); + down = 1; req->i.ifi_change |= IFF_UP; req->i.ifi_flags &= ~IFF_UP; + + sprintf(file, "/proc/%d/comm", ppid); + fd = open(file, O_RDONLY); + read(fd, comm, sizeof(comm)); + comm[strlen(comm) - 1] = '\0'; + printf("pid %d, comm %s calls ip to down device", ppid, comm); + if (*dev) + printf(" %s\n", *dev); + close(fd); + } else if (strcmp(*argv, "name") == 0) { NEXT_ARG(); *name = *argv; @@ -475,6 +493,8 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req, if (*dev) duparg2("dev", *argv); *dev = *argv; + if (down) + printf(" dev %s\n", *dev); } argc--; argv++; } -- Thanks!
Am 10.07.2013 11:42, schrieb Gao feng:
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); }
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } }
if (dev->flags & IFF_UP &&
Thanks!
There you go:
Thank you very much.
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8 [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8 [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918 [ 129.084426] Call Trace:
[ 129.084821] virbr0: port 6(veth5) entered disabled state
I can confirm it's the ip command disable the veth device now. but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git). use read /proc/<getppid>/comm to trace which command calls ip.
Will do. Thanks, //richard
ip/iplink.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
diff --git a/ip/iplink.c b/ip/iplink.c index dc98019..ec7ad87 100644 --- a/ip/iplink.c +++ b/ip/iplink.c @@ -282,6 +282,7 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req, int mtu = -1; int netns = -1; int vf = -1; + int down = 0; int numtxqueues = -1; int numrxqueues = -1;
@@ -293,8 +294,25 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req, req->i.ifi_change |= IFF_UP; req->i.ifi_flags |= IFF_UP; } else if (strcmp(*argv, "down") == 0) { + pid_t ppid = getppid(); + char file[30]; + char comm[20]; + int fd; + memset(file, 0, sizeof(file)); + memset(comm, 0, sizeof(comm)); + down = 1; req->i.ifi_change |= IFF_UP; req->i.ifi_flags &= ~IFF_UP; + + sprintf(file, "/proc/%d/comm", ppid); + fd = open(file, O_RDONLY); + read(fd, comm, sizeof(comm)); + comm[strlen(comm) - 1] = '\0'; + printf("pid %d, comm %s calls ip to down device", ppid, comm); + if (*dev) + printf(" %s\n", *dev); + close(fd); + } else if (strcmp(*argv, "name") == 0) { NEXT_ARG(); *name = *argv; @@ -475,6 +493,8 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req, if (*dev) duparg2("dev", *argv); *dev = *argv; + if (down) + printf(" dev %s\n", *dev); } argc--; argv++; }
Am 10.07.2013 11:42, schrieb Gao feng:
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); }
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } }
if (dev->flags & IFF_UP &&
Thanks!
There you go:
Thank you very much.
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8 [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8 [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918 [ 129.084426] Call Trace:
[ 129.084821] virbr0: port 6(veth5) entered disabled state
I can confirm it's the ip command disable the veth device now. but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git). use read /proc/<getppid>/comm to trace which command calls ip.
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this: 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down Now I have to urge to use a "Kantholz". ;-) Thanks, //richard
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
Am 10.07.2013 11:42, schrieb Gao feng:
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); }
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } }
if (dev->flags & IFF_UP &&
Thanks!
There you go:
Thank you very much.
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8 [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8 [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918 [ 129.084426] Call Trace:
[ 129.084821] virbr0: port 6(veth5) entered disabled state
I can confirm it's the ip command disable the veth device now. but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git). use read /proc/<getppid>/comm to trace which command calls ip.
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm... it's systemd... I have no idea now... :(
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
Am 10.07.2013 11:42, schrieb Gao feng:
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
> > Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
> I need to know why these two devices are down, I believe they were up, your bridge and default-net > looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5. I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :( I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c index d45e760..aed319b 100644 --- a/net/bridge/br_stp_if.c +++ b/net/bridge/br_stp_if.c @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p) p->state = BR_STATE_DISABLED; p->topology_change_ack = 0; p->config_pending = 0; - + dump_stack(); br_log_state(p); br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c index faebb39..9b1617b 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) { rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING); + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); }
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int old_flags) if (changes & IFF_UP) { if (dev->flags & IFF_UP) call_netdevice_notifiers(NETDEV_UP, dev); - else + else { + dump_stack(); call_netdevice_notifiers(NETDEV_DOWN, dev); + } }
if (dev->flags & IFF_UP &&
Thanks!
There you go:
Thank you very much.
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20 [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8 [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8 [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918 [ 129.084426] Call Trace:
[ 129.084821] virbr0: port 6(veth5) entered disabled state
I can confirm it's the ip command disable the veth device now. but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git). use read /proc/<getppid>/comm to trace which command calls ip.
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic. Thanks, //richard
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records.
Sure: https://bugzilla.novell.com/show_bug.cgi?id=829033 Thanks, //richard
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records.
It's good news we know what causes veth device down. :)
Am 12.07.2013 03:36, schrieb Gao feng:
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records.
It's good news we know what causes veth device down. :)
How does Fedora deal with veth devices? SUSE folks think that this is a more likely a libvirt issue and closed my bug report as invalid... Thanks, //richard
On Wed, Jul 17, 2013 at 11:33:22PM +0200, Richard Weinberger wrote:
Am 12.07.2013 03:36, schrieb Gao feng:
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote: > This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. > The log showed this: > > 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd > 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd > 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd > 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd > 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug > 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down > > Now I have to urge to use a "Kantholz". ;-) >
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records.
It's good news we know what causes veth device down. :)
How does Fedora deal with veth devices?
Well the udev script you mention above does not exist on Fedora and AFAIK there's no other udev script which runs 'ifconfig down' on NICs.
SUSE folks think that this is a more likely a libvirt issue and closed my bug report as invalid...
If you remove or modify the 77-network.rules file does it fix the problem. If so, then it is obviously not a libvirt issue. IMHO it is completely bogus for udev to be arbitrarily ifdown'ing any interface. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Richard, I think you should have cc'd the bug assignee when discussing this issue upstream. Adding him now... Regards, Jim Daniel P. Berrange wrote:
On Wed, Jul 17, 2013 at 11:33:22PM +0200, Richard Weinberger wrote:
Am 12.07.2013 03:36, schrieb Gao feng:
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
> On 07/11/2013 03:18 PM, Richard Weinberger wrote: > >> This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. >> The log showed this: >> >> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd >> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd >> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd >> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd >> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug >> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down >> >> Now I have to urge to use a "Kantholz". ;-) >> >> > hmmm... > > it's systemd... I have no idea now... :( > TBH it is not systemd's fault. OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records.
It's good news we know what causes veth device down. :)
How does Fedora deal with veth devices?
Well the udev script you mention above does not exist on Fedora and AFAIK there's no other udev script which runs 'ifconfig down' on NICs.
SUSE folks think that this is a more likely a libvirt issue and closed my bug report as invalid...
If you remove or modify the 77-network.rules file does it fix the problem. If so, then it is obviously not a libvirt issue. IMHO it is completely bogus for udev to be arbitrarily ifdown'ing any interface.
Daniel
Am 18.07.2013 16:50, schrieb Jim Fehlig:
Richard,
I think you should have cc'd the bug assignee when discussing this issue upstream. Adding him now...
Oh, sorry for that! I thought I did so after pointing Marius to the thread in the mailing list archive, but obviously I forgot. Thanks, //richard
Regards, Jim
Daniel P. Berrange wrote:
On Wed, Jul 17, 2013 at 11:33:22PM +0200, Richard Weinberger wrote:
Am 12.07.2013 03:36, schrieb Gao feng:
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
> Am 11.07.2013 11:42, schrieb Gao feng: > >> On 07/11/2013 03:18 PM, Richard Weinberger wrote: >> >>> This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. >>> The log showed this: >>> >>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd >>> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd >>> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd >>> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd >>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug >>> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down >>> >>> Now I have to urge to use a "Kantholz". ;-) >>> >>> >> hmmm... >> >> it's systemd... I have no idea now... :( >> > TBH it is not systemd's fault. > OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices. > Therefore systemd-udevd called ifup/down and other hotplug magic. > Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse to fix this ? Can you post a link to the bug here for the sake of archive records.
It's good news we know what causes veth device down. :)
How does Fedora deal with veth devices?
Well the udev script you mention above does not exist on Fedora and AFAIK there's no other udev script which runs 'ifconfig down' on NICs.
SUSE folks think that this is a more likely a libvirt issue and closed my bug report as invalid...
If you remove or modify the 77-network.rules file does it fix the problem. If so, then it is obviously not a libvirt issue. IMHO it is completely bogus for udev to be arbitrarily ifdown'ing any interface.
Daniel
Am 18.07.2013 17:02, schrieb Richard Weinberger:
Am 18.07.2013 16:50, schrieb Jim Fehlig:
Richard,
I think you should have cc'd the bug assignee when discussing this issue upstream. Adding him now...
Oh, sorry for that! I thought I did so after pointing Marius to the thread in the mailing list archive, but obviously I forgot.
Hi! This is a cleanup call (to stop dhcp-clients if any, remove state, ...) triggered by the remove event from kernel. IMO libvirt is missing an "udevadm settle" call between "remove foo; add foo" -- the "add" and "link up" calls in libvirt are done before udev finished the "remove" event processing. Gruesse / Regards, Marius Tomaschewski <mt@suse.de>, <mt@suse.com> -- SUSE LINUX Products GmbH, HRB 16746 (AG Nürnberg), GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, Maxfeldstraße 5, 90409 Nürnberg, Germany
On Thu, Jul 18, 2013 at 09:43:54PM +0200, Marius Tomaschewski wrote:
Am 18.07.2013 17:02, schrieb Richard Weinberger:
Am 18.07.2013 16:50, schrieb Jim Fehlig:
Richard,
I think you should have cc'd the bug assignee when discussing this issue upstream. Adding him now...
Oh, sorry for that! I thought I did so after pointing Marius to the thread in the mailing list archive, but obviously I forgot.
Hi!
This is a cleanup call (to stop dhcp-clients if any, remove state, ...) triggered by the remove event from kernel.
IMO libvirt is missing an "udevadm settle" call between "remove foo; add foo" -- the "add" and "link up" calls in libvirt are done before udev finished the "remove" event processing.
No, these ifup/ifdown udev rules should not be running for veth devices. They are not normal NICs - they are the same usage scenario as Xen vif devices or tap devices in QEMU. The suse 77-network.rules script is already whitelisting the vif* and tap* to avoid this behaviour for those devices and it should be doing the same for veth* devices too. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Am 19.07.2013 10:36, schrieb Daniel P. Berrange:
On Thu, Jul 18, 2013 at 09:43:54PM +0200, Marius Tomaschewski wrote:
Am 18.07.2013 17:02, schrieb Richard Weinberger:
Am 18.07.2013 16:50, schrieb Jim Fehlig:
Richard,
I think you should have cc'd the bug assignee when discussing this issue upstream. Adding him now...
Oh, sorry for that! I thought I did so after pointing Marius to the thread in the mailing list archive, but obviously I forgot.
Hi!
This is a cleanup call (to stop dhcp-clients if any, remove state, ...) triggered by the remove event from kernel.
IMO libvirt is missing an "udevadm settle" call between "remove foo; add foo" -- the "add" and "link up" calls in libvirt are done before udev finished the "remove" event processing.
No, these ifup/ifdown udev rules should not be running for veth devices. They are not normal NICs - they are the same usage scenario as Xen vif devices or tap devices in QEMU. The suse 77-network.rules script is already whitelisting the vif* and tap* to avoid this behaviour for those devices and it should be doing the same for veth* devices too.
The black list is there as an optimization for interfaces, which ifup creates itself (e.g. tap). Basically the blacklist can be also removed and ifup handles its nested calls then (by type, names are free). We've added vif* as optimization for XEN and when the user wants routed VM setup using ifup, then the user should specify a name from another namespace in the VM config. I can add veth and vnet to the backlist to optimize it once again, but libvirt needs an udevadm settle call as well or we have same situation in some time again -- just with a different name. Without a settle call, I will always consider this as a libvirt issue. Last but not least, the user can create own udev rules -- there are customers, where I know that they use own rules. Gruesse / Regards, Marius Tomaschewski <mt@suse.de>, <mt@suse.com> -- SUSE LINUX Products GmbH, HRB 16746 (AG Nürnberg), GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, Maxfeldstraße 5, 90409 Nürnberg, Germany
On Thu, Jul 11, 2013 at 09:18:24AM +0200, Richard Weinberger wrote:
This morning I've installed a wrapper around ip to show me the process tree upon ip link ... down is used. The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
Ahh, I had a feeling it was going to turn out to be udev's fault. No I wonder if there is some specific udev rule on your host causing this, since other people aren't see this behaviour. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (5)
-
Daniel P. Berrange -
Gao feng -
Jim Fehlig -
Marius Tomaschewski -
Richard Weinberger