2:45 a.m.
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled
libvirtd fails to
up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy.
1. Define a few containers (5 in my case)
2. Run "virsh autostart ..." on each one.
3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on
the host
side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
Thanks,
//richard
5:04 a.m.
Hi,
On 07/01/2013 03:45 PM, Richard Weinberger wrote:
Hi!
If you have multiple LXC containers with networking and the autostart feature enabled
libvirtd fails to
up some veth interfaces on the host side.
Most of the time only the first veth device is in state up, all others are down.
Reproducing is easy.
1. Define a few containers (5 in my case)
2. Run "virsh autostart ..." on each one.
3. stop/start libvirtd
You'll observe that all containers are running, but "ip a" will report on
the host
side that not all veth devices are up and are not usable within the containers.
This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
maybe you should wait some seconds for the starting of these containers.
Thanks!
3:45 p.m.
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
Hi,
On 07/01/2013 03:45 PM, Richard Weinberger wrote:
> Hi!
>
> If you have multiple LXC containers with networking and the autostart feature enabled
libvirtd fails to
> up some veth interfaces on the host side.
>
> Most of the time only the first veth device is in state up, all others are down.
>
> Reproducing is easy.
> 1. Define a few containers (5 in my case)
> 2. Run "virsh autostart ..." on each one.
> 3. stop/start libvirtd
>
> You'll observe that all containers are running, but "ip a" will report
on the host
> side that not all veth devices are up and are not usable within the containers.
>
> This is not userns related, just retested with libvirt of today.
I can not reproduce this problem on my test bed...
Strange.
maybe you should wait some seconds for the starting of these
containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all
of
my test machines.
run.sh creates six very minimal containers and enables autostart. Then it kills and
restarts libvirtd.
After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one:
for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done
All veths are always up.
Thanks,
//richard
8:36 p.m.
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng:
> Hi,
> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>> Hi!
>>
>> If you have multiple LXC containers with networking and the autostart feature
enabled libvirtd fails to
>> up some veth interfaces on the host side.
>>
>> Most of the time only the first veth device is in state up, all others are down.
>>
>> Reproducing is easy.
>> 1. Define a few containers (5 in my case)
>> 2. Run "virsh autostart ..." on each one.
>> 3. stop/start libvirtd
>>
>> You'll observe that all containers are running, but "ip a" will
report on the host
>> side that not all veth devices are up and are not usable within the containers.
>>
>> This is not userns related, just retested with libvirt of today.
>
> I can not reproduce this problem on my test bed...
Strange.
> maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all
of
my test machines.
run.sh creates six very minimal containers and enables autostart. Then it kills and
restarts libvirtd.
After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one:
for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done
All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh
Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
waiting a bit....
167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Thanks
5:22 a.m.
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
> Hi,
>
> Am 03.07.2013 12:04, schrieb Gao feng:
>> Hi,
>> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>>> Hi!
>>>
>>> If you have multiple LXC containers with networking and the autostart feature
enabled libvirtd fails to
>>> up some veth interfaces on the host side.
>>>
>>> Most of the time only the first veth device is in state up, all others are
down.
>>>
>>> Reproducing is easy.
>>> 1. Define a few containers (5 in my case)
>>> 2. Run "virsh autostart ..." on each one.
>>> 3. stop/start libvirtd
>>>
>>> You'll observe that all containers are running, but "ip a" will
report on the host
>>> side that not all veth devices are up and are not usable within the
containers.
>>>
>>> This is not userns related, just retested with libvirt of today.
>>
>> I can not reproduce this problem on my test bed...
>
> Strange.
>
>> maybe you should wait some seconds for the starting of these containers.
>
> Please see the attached shell script. Using it I'm able to trigger the issue on
all of
> my test machines.
> run.sh creates six very minimal containers and enables autostart. Then it kills and
restarts libvirtd.
> After the script is done you'll see that only one or two veth devices are up.
>
> On the over hand, if I start them manually using a command like this one:
> for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done
> All veths are always up.
>
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh
Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
waiting a bit....
167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0
state UP qlen 1000
45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0
state UP qlen 1000
47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN
qlen 1000
49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0
state UP qlen 1000
51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0
state UP qlen 1000
53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN
qlen 100
Thanks,
//richard
10:54 p.m.
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
Am 05.07.2013 03:36, schrieb Gao feng:
> On 07/05/2013 04:45 AM, Richard Weinberger wrote:
>> Hi,
>>
>> Am 03.07.2013 12:04, schrieb Gao feng:
>>> Hi,
>>> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>>>> Hi!
>>>>
>>>> If you have multiple LXC containers with networking and the autostart
feature enabled libvirtd fails to
>>>> up some veth interfaces on the host side.
>>>>
>>>> Most of the time only the first veth device is in state up, all others
are down.
>>>>
>>>> Reproducing is easy.
>>>> 1. Define a few containers (5 in my case)
>>>> 2. Run "virsh autostart ..." on each one.
>>>> 3. stop/start libvirtd
>>>>
>>>> You'll observe that all containers are running, but "ip a"
will report on the host
>>>> side that not all veth devices are up and are not usable within the
containers.
>>>>
>>>> This is not userns related, just retested with libvirt of today.
>>>
>>> I can not reproduce this problem on my test bed...
>>
>> Strange.
>>
>>> maybe you should wait some seconds for the starting of these containers.
>>
>> Please see the attached shell script. Using it I'm able to trigger the issue
on all of
>> my test machines.
>> run.sh creates six very minimal containers and enables autostart. Then it kills
and restarts libvirtd.
>> After the script is done you'll see that only one or two veth devices are
up.
>>
>> On the over hand, if I start them manually using a command like this one:
>> for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ;
done
>> All veths are always up.
>>
>
>
> I still can not reproduce even use your script.
>
> [root@Donkey-I5 Desktop]# ./run.sh
> Domain test-a defined from container_a.conf
>
> Domain test-a marked as autostarted
>
> Domain test-b defined from container_b.conf
>
> Domain test-b marked as autostarted
>
> Domain test-c defined from container_c.conf
>
> Domain test-c marked as autostarted
>
> Domain test-d defined from container_d.conf
>
> Domain test-d marked as autostarted
>
> Domain test-e defined from container_e.conf
>
> Domain test-e marked as autostarted
>
> Domain test-f defined from container_f.conf
>
> Domain test-f marked as autostarted
>
> 2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
> 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
> waiting a bit....
> 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
>
>
> Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN
qlen 1000
49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN
qlen 100
strange, I can not see veth related error message from your log.
seems like all of the veth devices of host had been up but for some reasons they become
down.
can you also show me the bridge information on your host?
brctl show virbr0
and your default net configuration
virsh -c lxc:/// net-dumpxml default
and your dmesg
Thanks!
8:11 a.m.
Am 08.07.2013 05:54, schrieb Gao feng:
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
> Am 05.07.2013 03:36, schrieb Gao feng:
>> On 07/05/2013 04:45 AM, Richard Weinberger wrote:
>>> Hi,
>>>
>>> Am 03.07.2013 12:04, schrieb Gao feng:
>>>> Hi,
>>>> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>>>>> Hi!
>>>>>
>>>>> If you have multiple LXC containers with networking and the autostart
feature enabled libvirtd fails to
>>>>> up some veth interfaces on the host side.
>>>>>
>>>>> Most of the time only the first veth device is in state up, all
others are down.
>>>>>
>>>>> Reproducing is easy.
>>>>> 1. Define a few containers (5 in my case)
>>>>> 2. Run "virsh autostart ..." on each one.
>>>>> 3. stop/start libvirtd
>>>>>
>>>>> You'll observe that all containers are running, but "ip
a" will report on the host
>>>>> side that not all veth devices are up and are not usable within the
containers.
>>>>>
>>>>> This is not userns related, just retested with libvirt of today.
>>>>
>>>> I can not reproduce this problem on my test bed...
>>>
>>> Strange.
>>>
>>>> maybe you should wait some seconds for the starting of these containers.
>>>
>>> Please see the attached shell script. Using it I'm able to trigger the
issue on all of
>>> my test machines.
>>> run.sh creates six very minimal containers and enables autostart. Then it
kills and restarts libvirtd.
>>> After the script is done you'll see that only one or two veth devices are
up.
>>>
>>> On the over hand, if I start them manually using a command like this one:
>>> for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg
; done
>>> All veths are always up.
>>>
>>
>>
>> I still can not reproduce even use your script.
>>
>> [root@Donkey-I5 Desktop]# ./run.sh
>> Domain test-a defined from container_a.conf
>>
>> Domain test-a marked as autostarted
>>
>> Domain test-b defined from container_b.conf
>>
>> Domain test-b marked as autostarted
>>
>> Domain test-c defined from container_c.conf
>>
>> Domain test-c marked as autostarted
>>
>> Domain test-d defined from container_d.conf
>>
>> Domain test-d marked as autostarted
>>
>> Domain test-e defined from container_e.conf
>>
>> Domain test-e marked as autostarted
>>
>> Domain test-f defined from container_f.conf
>>
>> Domain test-f marked as autostarted
>>
>> 2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
>> 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
>> waiting a bit....
>> 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>
>>
>> Can you post your libvirt debug log?
>
> Please see attached file.
>
> 43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state
DOWN qlen 1000
> 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virbr0 state UP qlen 1000
> 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state
DOWN qlen 100
>
strange, I can not see veth related error message from your log.
seems like all of the veth devices of host had been up but for some reasons they become
down.
I think libvirt has to do "ip link set dev vethX up".
Otherwise the device state is undefined.
can you also show me the bridge information on your host?
brctl show virbr0
bridge name bridge id STP enabled interfaces
virbr0 8000.2e63ddca07fa yes veth0
veth1
veth2
veth3
veth4
veth5
and your default net configuration
virsh -c lxc:/// net-dumpxml default
<network>
<name>default</name>
<uuid>a81b9f40-d279-4e96-8cec-6a906323f162</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr0' stp='on' delay='0' />
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254' />
</dhcp>
</ip>
</network>
Thanks,
//richard
8:20 p.m.
On 07/09/2013 09:11 PM, Richard Weinberger wrote:
Am 08.07.2013 05:54, schrieb Gao feng:
> On 07/05/2013 06:22 PM, Richard Weinberger wrote:
>> Am 05.07.2013 03:36, schrieb Gao feng:
>>> On 07/05/2013 04:45 AM, Richard Weinberger wrote:
>>>> Hi,
>>>>
>>>> Am 03.07.2013 12:04, schrieb Gao feng:
>>>>> Hi,
>>>>> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>>>>>> Hi!
>>>>>>
>>>>>> If you have multiple LXC containers with networking and the
autostart feature enabled libvirtd fails to
>>>>>> up some veth interfaces on the host side.
>>>>>>
>>>>>> Most of the time only the first veth device is in state up, all
others are down.
>>>>>>
>>>>>> Reproducing is easy.
>>>>>> 1. Define a few containers (5 in my case)
>>>>>> 2. Run "virsh autostart ..." on each one.
>>>>>> 3. stop/start libvirtd
>>>>>>
>>>>>> You'll observe that all containers are running, but "ip
a" will report on the host
>>>>>> side that not all veth devices are up and are not usable within
the containers.
>>>>>>
>>>>>> This is not userns related, just retested with libvirt of today.
>>>>>
>>>>> I can not reproduce this problem on my test bed...
>>>>
>>>> Strange.
>>>>
>>>>> maybe you should wait some seconds for the starting of these
containers.
>>>>
>>>> Please see the attached shell script. Using it I'm able to trigger
the issue on all of
>>>> my test machines.
>>>> run.sh creates six very minimal containers and enables autostart. Then it
kills and restarts libvirtd.
>>>> After the script is done you'll see that only one or two veth devices
are up.
>>>>
>>>> On the over hand, if I start them manually using a command like this
one:
>>>> for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start
test-$cfg ; done
>>>> All veths are always up.
>>>>
>>>
>>>
>>> I still can not reproduce even use your script.
>>>
>>> [root@Donkey-I5 Desktop]# ./run.sh
>>> Domain test-a defined from container_a.conf
>>>
>>> Domain test-a marked as autostarted
>>>
>>> Domain test-b defined from container_b.conf
>>>
>>> Domain test-b marked as autostarted
>>>
>>> Domain test-c defined from container_c.conf
>>>
>>> Domain test-c marked as autostarted
>>>
>>> Domain test-d defined from container_d.conf
>>>
>>> Domain test-d marked as autostarted
>>>
>>> Domain test-e defined from container_e.conf
>>>
>>> Domain test-e marked as autostarted
>>>
>>> Domain test-f defined from container_f.conf
>>>
>>> Domain test-f marked as autostarted
>>>
>>> 2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
>>> 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
>>> waiting a bit....
>>> 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>>
>>>
>>> Can you post your libvirt debug log?
>>
>> Please see attached file.
>>
>> 43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0
state DOWN qlen 1000
>> 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>> 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0
state DOWN qlen 100
>>
>
> strange, I can not see veth related error message from your log.
>
> seems like all of the veth devices of host had been up but for some reasons they
become down.
I think libvirt has to do "ip link set dev vethX up".
Otherwise the device state is undefined.
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are
down.
I need to know why these two devices are down, I believe they were up, your bridge and
default-net
looks good. So please show me your kernel message (dmesg), maybe it can give us some
useful information.
Thanks
1 a.m.
Am 10.07.2013 03:20, schrieb Gao feng:
On 07/09/2013 09:11 PM, Richard Weinberger wrote:
> Am 08.07.2013 05:54, schrieb Gao feng:
>> On 07/05/2013 06:22 PM, Richard Weinberger wrote:
>>> Am 05.07.2013 03:36, schrieb Gao feng:
>>>> On 07/05/2013 04:45 AM, Richard Weinberger wrote:
>>>>> Hi,
>>>>>
>>>>> Am 03.07.2013 12:04, schrieb Gao feng:
>>>>>> Hi,
>>>>>> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>>>>>>> Hi!
>>>>>>>
>>>>>>> If you have multiple LXC containers with networking and the
autostart feature enabled libvirtd fails to
>>>>>>> up some veth interfaces on the host side.
>>>>>>>
>>>>>>> Most of the time only the first veth device is in state up,
all others are down.
>>>>>>>
>>>>>>> Reproducing is easy.
>>>>>>> 1. Define a few containers (5 in my case)
>>>>>>> 2. Run "virsh autostart ..." on each one.
>>>>>>> 3. stop/start libvirtd
>>>>>>>
>>>>>>> You'll observe that all containers are running, but
"ip a" will report on the host
>>>>>>> side that not all veth devices are up and are not usable
within the containers.
>>>>>>>
>>>>>>> This is not userns related, just retested with libvirt of
today.
>>>>>>
>>>>>> I can not reproduce this problem on my test bed...
>>>>>
>>>>> Strange.
>>>>>
>>>>>> maybe you should wait some seconds for the starting of these
containers.
>>>>>
>>>>> Please see the attached shell script. Using it I'm able to
trigger the issue on all of
>>>>> my test machines.
>>>>> run.sh creates six very minimal containers and enables autostart.
Then it kills and restarts libvirtd.
>>>>> After the script is done you'll see that only one or two veth
devices are up.
>>>>>
>>>>> On the over hand, if I start them manually using a command like this
one:
>>>>> for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start
test-$cfg ; done
>>>>> All veths are always up.
>>>>>
>>>>
>>>>
>>>> I still can not reproduce even use your script.
>>>>
>>>> [root@Donkey-I5 Desktop]# ./run.sh
>>>> Domain test-a defined from container_a.conf
>>>>
>>>> Domain test-a marked as autostarted
>>>>
>>>> Domain test-b defined from container_b.conf
>>>>
>>>> Domain test-b marked as autostarted
>>>>
>>>> Domain test-c defined from container_c.conf
>>>>
>>>> Domain test-c marked as autostarted
>>>>
>>>> Domain test-d defined from container_d.conf
>>>>
>>>> Domain test-d marked as autostarted
>>>>
>>>> Domain test-e defined from container_e.conf
>>>>
>>>> Domain test-e marked as autostarted
>>>>
>>>> Domain test-f defined from container_f.conf
>>>>
>>>> Domain test-f marked as autostarted
>>>>
>>>> 2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
>>>> 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
>>>> waiting a bit....
>>>> 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>>
>>>>
>>>> Can you post your libvirt debug log?
>>>
>>> Please see attached file.
>>>
>>> 43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master
virbr0 state DOWN qlen 1000
>>> 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master
virbr0 state DOWN qlen 100
>>>
>>
>> strange, I can not see veth related error message from your log.
>>
>> seems like all of the veth devices of host had been up but for some reasons they
become down.
>
> I think libvirt has to do "ip link set dev vethX up".
> Otherwise the device state is undefined.
>
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are
down.
Where does libvirt up the devices? The debug log does not contain any "ip link set
dev XXX up" commands.
Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
I need to know why these two devices are down, I believe they were
up, your bridge and default-net
looks good. So please show me your kernel message (dmesg), maybe it can give us some
useful information.
This time veth4 and 5 are down.
---cut---
[ 4.167119] systemd[1]: Starting File System Check on Root Device...
[ 5.150740] mount (725) used greatest stack depth: 4616 bytes left
[ 5.397652] systemd-udevd[755]: starting version 195
[ 6.155648] EXT4-fs (vda2): re-mounted. Opts: acl,user_xattr
[ 6.369755] systemd-journald[730]: Received SIGUSR1
[ 6.877588] Adding 690172k swap on /dev/vda1. Priority:-1 extents:1 across:690172k
[ 8.205295] ip (1446) used greatest stack depth: 4584 bytes left
[ 9.397258] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 9.397275] 8021q: adding VLAN 0 to HW filter on device eth0
[ 9.397733] ip (1690) used greatest stack depth: 4456 bytes left
[ 11.399834] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 11.401711] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 41.179748] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready
[ 42.126303] cgroup: libvirtd (2736) created nested cgroup for controller
"memory" which has incomplete hierarchy support. Nested cgroups may change
behavior in the future.
[ 42.126307] cgroup: "memory" requires setting use_hierarchy to 1 on the
root.
[ 42.126599] cgroup: libvirtd (2736) created nested cgroup for controller
"blkio" which has incomplete hierarchy support. Nested cgroups may change
behavior in the future.
[ 42.145511] ip (2843) used greatest stack depth: 4136 bytes left
[ 42.149231] device veth0 entered promiscuous mode
[ 42.152167] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[ 42.559264] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[ 42.559336] virbr0: topology change detected, propagating
[ 42.559343] virbr0: port 1(veth0) entered forwarding state
[ 42.559359] virbr0: port 1(veth0) entered forwarding state
[ 42.559398] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready
[ 42.702130] ip (3070) used greatest stack depth: 4088 bytes left
[ 42.708164] device veth1 entered promiscuous mode
[ 42.712190] IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready
[ 42.712197] virbr0: topology change detected, propagating
[ 42.712202] virbr0: port 2(veth1) entered forwarding state
[ 42.712212] virbr0: port 2(veth1) entered forwarding state
[ 43.147155] virbr0: port 2(veth1) entered disabled state
[ 43.225250] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
[ 43.225322] virbr0: topology change detected, propagating
[ 43.225328] virbr0: port 2(veth1) entered forwarding state
[ 43.225358] virbr0: port 2(veth1) entered forwarding state
[ 43.325556] ip (3293) used greatest stack depth: 4064 bytes left
[ 43.330133] device veth2 entered promiscuous mode
[ 43.334201] IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready
[ 43.334208] virbr0: topology change detected, propagating
[ 43.334214] virbr0: port 3(veth2) entered forwarding state
[ 43.334224] virbr0: port 3(veth2) entered forwarding state
[ 43.613184] IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready
[ 43.767050] ip (3521) used greatest stack depth: 3992 bytes left
[ 43.773832] device veth3 entered promiscuous mode
[ 43.778169] IPv6: ADDRCONF(NETDEV_UP): veth3: link is not ready
[ 43.778177] virbr0: topology change detected, propagating
[ 43.778182] virbr0: port 4(veth3) entered forwarding state
[ 43.778193] virbr0: port 4(veth3) entered forwarding state
[ 44.076299] IPv6: ADDRCONF(NETDEV_CHANGE): veth3: link becomes ready
[ 44.153214] device veth4 entered promiscuous mode
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
[ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
[ 44.473400] virbr0: topology change detected, propagating
[ 44.473407] virbr0: port 5(veth4) entered forwarding state
[ 44.473423] virbr0: port 5(veth4) entered forwarding state
[ 44.566186] device veth5 entered promiscuous mode
[ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
[ 44.571243] virbr0: topology change detected, propagating
[ 44.571250] virbr0: port 6(veth5) entered forwarding state
[ 44.571261] virbr0: port 6(veth5) entered forwarding state
[ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
[ 45.000580] virbr0: port 5(veth4) entered disabled state
[ 45.348548] virbr0: port 6(veth5) entered disabled state
[ 45.348837] ip (4349) used greatest stack depth: 3944 bytes left
---cut---
Thanks,
//richard
2:03 a.m.
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>
> Yes,actually libvirt did up the veth devices, that's why only veth2& veth5
are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set
dev XXX up" commands.
Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
> I need to know why these two devices are down, I believe they
were up, your bridge and default-net
> looks good. So please show me your kernel message (dmesg), maybe it can give us some
useful information.
This time veth4 and 5 are down.
---cut---
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
[ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
[ 44.473400] virbr0: topology change detected, propagating
[ 44.473407] virbr0: port 5(veth4) entered forwarding state
[ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
[ 44.566186] device veth5 entered promiscuous mode
[ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
[ 44.571243] virbr0: topology change detected, propagating
[ 44.571250] virbr0: port 6(veth5) entered forwarding state
[ 44.571261] virbr0: port 6(veth5) entered forwarding state
[ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
[ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
[ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5.
I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :(
I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
index d45e760..aed319b 100644
--- a/net/bridge/br_stp_if.c
+++ b/net/bridge/br_stp_if.c
@@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
p->state = BR_STATE_DISABLED;
p->topology_change_ack = 0;
p->config_pending = 0;
-
+ dump_stack();
br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c
index faebb39..9b1617b 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) {
rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
+ dump_stack();
call_netdevice_notifiers(NETDEV_DOWN, dev);
}
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int
old_flags)
if (changes & IFF_UP) {
if (dev->flags & IFF_UP)
call_netdevice_notifiers(NETDEV_UP, dev);
- else
+ else {
+ dump_stack();
call_netdevice_notifiers(NETDEV_DOWN, dev);
+ }
}
if (dev->flags & IFF_UP &&
Thanks!
2:23 a.m.
Am 10.07.2013 09:03, schrieb Gao feng:
On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>>
>> Yes,actually libvirt did up the veth devices, that's why only veth2&
veth5 are down.
>
> Where does libvirt up the devices? The debug log does not contain any "ip link
set dev XXX up" commands.
> Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
>
virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
>> I need to know why these two devices are down, I believe they
were up, your bridge and default-net
>> looks good. So please show me your kernel message (dmesg), maybe it can give us
some useful information.
>
> This time veth4 and 5 are down.
>
> ---cut---
> [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
> [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
> [ 44.473400] virbr0: topology change detected, propagating
> [ 44.473407] virbr0: port 5(veth4) entered forwarding state
> [ 44.473423] virbr0: port 5(veth4) entered forwarding state
veth4 were up here
> [ 44.566186] device veth5 entered promiscuous mode
> [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
> [ 44.571243] virbr0: topology change detected, propagating
> [ 44.571250] virbr0: port 6(veth5) entered forwarding state
> [ 44.571261] virbr0: port 6(veth5) entered forwarding state
> [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
> [ 45.000580] virbr0: port 5(veth4) entered disabled state
and then it became down.
> [ 45.348548] virbr0: port 6(veth5) entered disabled state
So, Some places disable the veth4 and veth5.
I don't know in which case these two devices will be disabled.
I still can't reproduce this problem in my test bed :(
I need more information to analyse why these two device being disabled.
So, can you run kernel with the below debug patch?
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
index d45e760..aed319b 100644
--- a/net/bridge/br_stp_if.c
+++ b/net/bridge/br_stp_if.c
@@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
p->state = BR_STATE_DISABLED;
p->topology_change_ack = 0;
p->config_pending = 0;
-
+ dump_stack();
br_log_state(p);
br_ifinfo_notify(RTM_NEWLINK, p);
diff --git a/net/core/dev.c b/net/core/dev.c
index faebb39..9b1617b 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
list_for_each_entry(dev, head, unreg_list) {
rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
+ dump_stack();
call_netdevice_notifiers(NETDEV_DOWN, dev);
}
@@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int
old_flags)
if (changes & IFF_UP) {
if (dev->flags & IFF_UP)
call_netdevice_notifiers(NETDEV_UP, dev);
- else
+ else {
+ dump_stack();
call_netdevice_notifiers(NETDEV_DOWN, dev);
+ }
}
if (dev->flags & IFF_UP &&
Thanks!
There you go:
[ 2.517243] systemd-journald[729]: Received SIGUSR1
[ 3.427222] Adding 690172k swap on /dev/vda1. Priority:-1 extents:1 across:690172k
[ 5.703781] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 5.703796] 8021q: adding VLAN 0 to HW filter on device eth0
[ 7.708579] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 7.709232] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 126.002595] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready
[ 126.450081] cgroup: libvirtd (2763) created nested cgroup for controller
"memory" which has incomplete hierarchy support. Nested cgroups may change
behavior in the future.
[ 126.450085] cgroup: "memory" requires setting use_hierarchy to 1 on the
root.
[ 126.450365] cgroup: libvirtd (2763) created nested cgroup for controller
"blkio" which has incomplete hierarchy support. Nested cgroups may change
behavior in the future.
[ 126.463191] device veth0 entered promiscuous mode
[ 126.468207] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[ 126.468216] virbr0: topology change detected, propagating
[ 126.468222] virbr0: port 1(veth0) entered forwarding state
[ 126.468237] virbr0: port 1(veth0) entered forwarding state
[ 126.468265] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready
[ 126.689269] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[ 126.786773] ip (3094) used greatest stack depth: 4072 bytes left
[ 126.793169] device veth1 entered promiscuous mode
[ 126.799334] IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready
[ 126.799342] virbr0: topology change detected, propagating
[ 126.799348] virbr0: port 2(veth1) entered forwarding state
[ 126.799360] virbr0: port 2(veth1) entered forwarding state
[ 126.903181] CPU: 0 PID: 769 Comm: kworker/0:2 Not tainted 3.10.0+ #20
[ 126.903187] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 126.903226] Workqueue: events linkwatch_event
[ 126.903231] ffff88003bee2800 ffff88003d6b1c38 ffffffff8172b2a6 ffff88003d6b1c78
[ 126.903235] ffffffff81666d8e 0000000100000000 ffff88003bc11800 ffff88003bee2800
[ 126.903240] ffff88003cf5d000 ffff88003cf5d000 0000000000000000 ffff88003d6b1ca8
[ 126.903244] Call Trace:
[ 126.903268] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 126.903288] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100
[ 126.903294] [<ffffffff81663e38>] br_port_carrier_check+0xd8/0xe0
[ 126.903299] [<ffffffff816655d8>] br_device_event+0x168/0x200
[ 126.903326] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70
[ 126.903332] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10
[ 126.903337] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20
[ 126.903347] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60
[ 126.903352] [<ffffffff815484b2>] netdev_state_change+0x22/0x40
[ 126.903357] [<ffffffff8155bd47>] linkwatch_do_dev+0x37/0x60
[ 126.903362] [<ffffffff8155bf56>] __linkwatch_run_queue+0xe6/0x170
[ 126.903367] [<ffffffff8155c000>] linkwatch_event+0x20/0x30
[ 126.903373] [<ffffffff810567bf>] process_one_work+0x17f/0x420
[ 126.903377] [<ffffffff81056ea9>] worker_thread+0x119/0x370
[ 126.903382] [<ffffffff81056d90>] ? rescuer_thread+0x2f0/0x2f0
[ 126.903388] [<ffffffff8105d42b>] kthread+0xbb/0xc0
[ 126.903394] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50
[ 126.903404] [<ffffffff8173326c>] ret_from_fork+0x7c/0xb0
[ 126.903410] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50
[ 126.903414] virbr0: port 2(veth1) entered disabled state
[ 127.038275] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
[ 127.038360] virbr0: topology change detected, propagating
[ 127.038368] virbr0: port 2(veth1) entered forwarding state
[ 127.038382] virbr0: port 2(veth1) entered forwarding state
[ 127.124475] ip (3296) used greatest stack depth: 4016 bytes left
[ 127.129185] device veth2 entered promiscuous mode
[ 127.134253] IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready
[ 127.134261] virbr0: topology change detected, propagating
[ 127.134267] virbr0: port 3(veth2) entered forwarding state
[ 127.134278] virbr0: port 3(veth2) entered forwarding state
[ 127.325301] IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready
[ 127.441199] device veth3 entered promiscuous mode
[ 127.445205] IPv6: ADDRCONF(NETDEV_UP): veth3: link is not ready
[ 127.445213] virbr0: topology change detected, propagating
[ 127.445218] virbr0: port 4(veth3) entered forwarding state
[ 127.445228] virbr0: port 4(veth3) entered forwarding state
[ 127.703335] IPv6: ADDRCONF(NETDEV_CHANGE): veth3: link becomes ready
[ 127.814220] device veth4 entered promiscuous mode
[ 127.818206] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
[ 127.818215] virbr0: topology change detected, propagating
[ 127.818221] virbr0: port 5(veth4) entered forwarding state
[ 127.818232] virbr0: port 5(veth4) entered forwarding state
[ 127.903178] CPU: 1 PID: 352 Comm: kworker/1:1 Not tainted 3.10.0+ #20
[ 127.903183] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 127.903191] Workqueue: events linkwatch_event
[ 127.903194] ffff88003bee2800 ffff88003db27c38 ffffffff8172b2a6 ffff88003db27c78
[ 127.903198] ffffffff81666d8e 0000000100000000 ffff8800374b0400 ffff88003bee2800
[ 127.903201] ffff88003b39e000 ffff88003b39e000 0000000000000000 ffff88003db27ca8
[ 127.903204] Call Trace:
[ 127.903211] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 127.903216] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100
[ 127.903221] [<ffffffff81663e38>] br_port_carrier_check+0xd8/0xe0
[ 127.903224] [<ffffffff816655d8>] br_device_event+0x168/0x200
[ 127.903230] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70
[ 127.903234] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10
[ 127.903238] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20
[ 127.903242] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60
[ 127.903245] [<ffffffff815484b2>] netdev_state_change+0x22/0x40
[ 127.903249] [<ffffffff8155bd47>] linkwatch_do_dev+0x37/0x60
[ 127.903253] [<ffffffff8155bf56>] __linkwatch_run_queue+0xe6/0x170
[ 127.903256] [<ffffffff8155c000>] linkwatch_event+0x20/0x30
[ 127.903260] [<ffffffff810567bf>] process_one_work+0x17f/0x420
[ 127.903263] [<ffffffff81056ea9>] worker_thread+0x119/0x370
[ 127.903266] [<ffffffff81056d90>] ? rescuer_thread+0x2f0/0x2f0
[ 127.903270] [<ffffffff8105d42b>] kthread+0xbb/0xc0
[ 127.903274] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50
[ 127.903278] [<ffffffff8173326c>] ret_from_fork+0x7c/0xb0
[ 127.903283] [<ffffffff8105d370>] ? __kthread_unpark+0x50/0x50
[ 127.903287] virbr0: port 5(veth4) entered disabled state
[ 128.089226] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
[ 128.089281] virbr0: topology change detected, propagating
[ 128.089286] virbr0: port 5(veth4) entered forwarding state
[ 128.089297] virbr0: port 5(veth4) entered forwarding state
[ 128.098859] CPU: 1 PID: 3963 Comm: ip Not tainted 3.10.0+ #20
[ 128.098864] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 128.098867] ffff88003b21c000 ffff8800374cd798 ffffffff8172b2a6 ffff8800374cd7b8
[ 128.098871] ffffffff8154be04 ffff88003b21c000 0000000000001103 ffff8800374cd7e8
[ 128.098874] ffffffff8154be60 0000000000000010 ffff88003b21c000 ffff8800374cd918
[ 128.098877] Call Trace:
[ 128.098885] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 128.098890] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90
[ 128.098894] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 128.098898] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 128.098921] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 128.098937] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 128.098941] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 128.098944] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 128.098948] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 128.098964] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 128.098971] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 128.098974] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 128.098983] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 128.098987] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 128.098990] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 128.098994] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 128.099000] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 128.099015] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 128.099067] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 128.099072] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 128.099080] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 128.099091] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 128.099095] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 128.099098] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 128.099101] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 128.099108] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 128.099259] CPU: 1 PID: 3963 Comm: ip Not tainted 3.10.0+ #20
[ 128.099262] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 128.099263] ffff88003bee2800 ffff8800374cd698 ffffffff8172b2a6 ffff8800374cd6d8
[ 128.099267] ffffffff81666d8e ffff88003ccd1008 0000000000000002 ffff88003b10f400
[ 128.099270] ffff88003bee2800 ffff88003b21c000 0000000000000000 ffff8800374cd718
[ 128.099273] Call Trace:
[ 128.099277] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 128.099282] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100
[ 128.099285] [<ffffffff81665668>] br_device_event+0x1f8/0x200
[ 128.099289] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400
[ 128.099295] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70
[ 128.099299] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10
[ 128.099302] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20
[ 128.099306] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60
[ 128.099310] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90
[ 128.099313] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 128.099317] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 128.099320] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 128.099324] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 128.099328] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 128.099331] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 128.099335] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 128.099339] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 128.099343] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 128.099346] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 128.099350] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 128.099353] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 128.099357] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 128.099360] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 128.099364] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 128.099368] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 128.099371] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 128.099376] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 128.099380] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 128.099384] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 128.099387] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 128.099390] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 128.099393] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 128.099396] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 128.099399] virbr0: port 3(veth2) entered disabled state
[ 128.103318] ip (3963) used greatest stack depth: 3864 bytes left
[ 128.203201] device veth5 entered promiscuous mode
[ 128.207856] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
[ 128.207865] virbr0: topology change detected, propagating
[ 128.207872] virbr0: port 6(veth5) entered forwarding state
[ 128.207885] virbr0: port 6(veth5) entered forwarding state
[ 128.454311] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
[ 128.564812] CPU: 1 PID: 4299 Comm: ip Not tainted 3.10.0+ #20
[ 128.564817] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 128.564819] ffff88003b21e000 ffff880037531798 ffffffff8172b2a6 ffff8800375317b8
[ 128.564823] ffffffff8154be04 ffff88003b21e000 0000000000001103 ffff8800375317e8
[ 128.564826] ffffffff8154be60 0000000000000010 ffff88003b21e000 ffff880037531918
[ 128.564830] Call Trace:
[ 128.564838] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 128.564843] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90
[ 128.564846] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 128.564851] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 128.564856] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 128.564860] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 128.564864] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 128.564878] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 128.564882] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 128.564888] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 128.564893] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 128.564896] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 128.564901] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 128.564904] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 128.564907] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 128.564911] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 128.564916] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 128.564920] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 128.564925] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 128.564929] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 128.564934] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 128.564939] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 128.564942] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 128.564945] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 128.564948] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 128.564953] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 128.565135] CPU: 1 PID: 4299 Comm: ip Not tainted 3.10.0+ #20
[ 128.565137] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 128.565138] ffff88003bee2800 ffff880037531698 ffffffff8172b2a6 ffff8800375316d8
[ 128.565142] ffffffff81666d8e ffff88003b17e808 0000000000000002 ffff88003b394400
[ 128.565145] ffff88003bee2800 ffff88003b21e000 0000000000000000 ffff880037531718
[ 128.565148] Call Trace:
[ 128.565152] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 128.565156] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100
[ 128.565162] [<ffffffff81665668>] br_device_event+0x1f8/0x200
[ 128.565166] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400
[ 128.565170] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70
[ 128.565174] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10
[ 128.565177] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20
[ 128.565181] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60
[ 128.565185] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90
[ 128.565190] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 128.565193] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 128.565197] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 128.565201] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 128.565205] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 128.565209] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 128.565213] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 128.565218] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 128.565221] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 128.565225] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 128.565229] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 128.565232] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 128.565235] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 128.565239] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 128.565243] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 128.565246] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 128.565250] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 128.565254] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 128.565258] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 128.565263] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 128.565266] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 128.565269] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 128.565272] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 128.565275] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 128.565280] virbr0: port 4(veth3) entered disabled state
[ 128.796890] CPU: 1 PID: 4432 Comm: ip Not tainted 3.10.0+ #20
[ 128.796896] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 128.796899] ffff88003b39e000 ffff88003779f798 ffffffff8172b2a6 ffff88003779f7b8
[ 128.796905] ffffffff8154be04 ffff88003b39e000 0000000000001103 ffff88003779f7e8
[ 128.796920] ffffffff8154be60 0000000000000010 ffff88003b39e000 ffff88003779f918
[ 128.796924] Call Trace:
[ 128.796931] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 128.796936] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90
[ 128.796940] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 128.796945] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 128.796950] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 128.796955] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 128.796959] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 128.796962] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 128.796966] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 128.796971] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 128.796976] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 128.796979] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 128.796983] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 128.796986] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 128.796990] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 128.796993] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 128.796998] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 128.797003] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 128.797013] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 128.797049] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 128.797054] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 128.797059] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 128.797063] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 128.797066] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 128.797069] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 128.797073] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 128.797182] CPU: 1 PID: 4432 Comm: ip Not tainted 3.10.0+ #20
[ 128.797184] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 128.797186] ffff88003bee2800 ffff88003779f698 ffffffff8172b2a6 ffff88003779f6d8
[ 128.797189] ffffffff81666d8e ffff88003b281808 0000000000000002 ffff8800374b0400
[ 128.797192] ffff88003bee2800 ffff88003b39e000 0000000000000000 ffff88003779f718
[ 128.797195] Call Trace:
[ 128.797199] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 128.797203] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100
[ 128.797207] [<ffffffff81665668>] br_device_event+0x1f8/0x200
[ 128.797211] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400
[ 128.797215] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70
[ 128.797219] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10
[ 128.797223] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20
[ 128.797227] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60
[ 128.797230] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90
[ 128.797234] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 128.797237] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 128.797241] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 128.797245] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 128.797248] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 128.797251] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 128.797255] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 128.797260] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 128.797263] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 128.797266] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 128.797270] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 128.797273] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 128.797277] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 128.797280] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 128.797284] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 128.797288] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 128.797291] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 128.797301] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 128.797306] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 128.797310] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 128.797313] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 128.797316] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 128.797319] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 128.797322] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 128.797325] virbr0: port 5(veth4) entered disabled state
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
[ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8
[ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8
[ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918
[ 129.084426] Call Trace:
[ 129.084433] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 129.084438] [<ffffffff8154be04>] __dev_notify_flags+0x74/0x90
[ 129.084442] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 129.084446] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 129.084451] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 129.084456] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 129.084460] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 129.084463] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 129.084467] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 129.084473] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 129.084477] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 129.084481] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 129.084485] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 129.084488] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 129.084492] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 129.084495] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 129.084501] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 129.084505] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 129.084509] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 129.084513] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 129.084518] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 129.084523] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 129.084526] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 129.084530] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 129.084533] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 129.084537] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 129.084672] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
[ 129.084674] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 129.084676] ffff88003bee2800 ffff88003ce7f698 ffffffff8172b2a6 ffff88003ce7f6d8
[ 129.084679] ffffffff81666d8e ffff88003764c008 0000000000000002 ffff8800374b1400
[ 129.084682] ffff88003bee2800 ffff88003760d000 0000000000000000 ffff88003ce7f718
[ 129.084685] Call Trace:
[ 129.084689] [<ffffffff8172b2a6>] dump_stack+0x19/0x1b
[ 129.084694] [<ffffffff81666d8e>] br_stp_disable_port+0x5e/0x100
[ 129.084697] [<ffffffff81665668>] br_device_event+0x1f8/0x200
[ 129.084701] [<ffffffff8162b6ce>] ? addrconf_notify+0xbe/0x400
[ 129.084706] [<ffffffff8106298d>] notifier_call_chain+0x4d/0x70
[ 129.084710] [<ffffffff81062df9>] __raw_notifier_call_chain+0x9/0x10
[ 129.084713] [<ffffffff81062e11>] raw_notifier_call_chain+0x11/0x20
[ 129.084717] [<ffffffff81548151>] call_netdevice_notifiers+0x31/0x60
[ 129.084721] [<ffffffff8154be11>] __dev_notify_flags+0x81/0x90
[ 129.084724] [<ffffffff8154be60>] dev_change_flags+0x40/0x70
[ 129.084728] [<ffffffff81559a31>] do_setlink+0x351/0x980
[ 129.084731] [<ffffffff8162731e>] ? inet6_fill_ifla6_attrs+0x2fe/0x330
[ 129.084735] [<ffffffff813a2891>] ? nla_parse+0x31/0xe0
[ 129.084739] [<ffffffff8155a79a>] rtnl_newlink+0x29a/0x5a0
[ 129.084742] [<ffffffff81557fcc>] ? rtnl_fill_ifinfo+0x7dc/0xa60
[ 129.084746] [<ffffffff8155856f>] rtnetlink_rcv_msg+0x9f/0x240
[ 129.084761] [<ffffffff81122c0b>] ? __kmalloc_node_track_caller+0x3b/0x250
[ 129.084765] [<ffffffff8153bc2d>] ? __alloc_skb+0x7d/0x290
[ 129.084768] [<ffffffff815584d0>] ? __rtnl_unlock+0x20/0x20
[ 129.084772] [<ffffffff81574df9>] netlink_rcv_skb+0xa9/0xc0
[ 129.084776] [<ffffffff815584a0>] rtnetlink_rcv+0x20/0x30
[ 129.084779] [<ffffffff81573779>] netlink_unicast+0x159/0x1d0
[ 129.084782] [<ffffffff81573eaa>] netlink_sendmsg+0x2ba/0x370
[ 129.084787] [<ffffffff81534301>] sock_sendmsg+0xa1/0xd0
[ 129.084790] [<ffffffff810dad19>] ? find_get_page+0x19/0xa0
[ 129.084794] [<ffffffff81385e21>] ? cpumask_any_but+0x31/0x40
[ 129.084798] [<ffffffff8153467c>] ___sys_sendmsg+0x34c/0x360
[ 129.084802] [<ffffffff81100d72>] ? handle_mm_fault+0x152/0x200
[ 129.084806] [<ffffffff8102f5c4>] ? __do_page_fault+0x1c4/0x4b0
[ 129.084809] [<ffffffff81106b8b>] ? do_brk+0x1bb/0x320
[ 129.084812] [<ffffffff81535704>] __sys_sendmsg+0x44/0x80
[ 129.084815] [<ffffffff8153574d>] SyS_sendmsg+0xd/0x20
[ 129.084818] [<ffffffff81733312>] system_call_fastpath+0x16/0x1b
[ 129.084821] virbr0: port 6(veth5) entered disabled state
Thanks,
//richard
4:42 a.m.
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
Am 10.07.2013 09:03, schrieb Gao feng:
> On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>
>>>
>>> Yes,actually libvirt did up the veth devices, that's why only veth2&
veth5 are down.
>>
>> Where does libvirt up the devices? The debug log does not contain any "ip
link set dev XXX up" commands.
>> Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
>>
>
> virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
Ahhhh, it's using an ioctl().
>>> I need to know why these two devices are down, I believe they were up, your
bridge and default-net
>>> looks good. So please show me your kernel message (dmesg), maybe it can give
us some useful information.
>>
>> This time veth4 and 5 are down.
>>
>> ---cut---
>
>> [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
>> [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
>> [ 44.473400] virbr0: topology change detected, propagating
>> [ 44.473407] virbr0: port 5(veth4) entered forwarding state
>> [ 44.473423] virbr0: port 5(veth4) entered forwarding state
>
> veth4 were up here
>
>> [ 44.566186] device veth5 entered promiscuous mode
>> [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
>> [ 44.571243] virbr0: topology change detected, propagating
>> [ 44.571250] virbr0: port 6(veth5) entered forwarding state
>> [ 44.571261] virbr0: port 6(veth5) entered forwarding state
>> [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
>> [ 45.000580] virbr0: port 5(veth4) entered disabled state
>
> and then it became down.
>
>> [ 45.348548] virbr0: port 6(veth5) entered disabled state
>
> So, Some places disable the veth4 and veth5.
> I don't know in which case these two devices will be disabled.
>
> I still can't reproduce this problem in my test bed :(
> I need more information to analyse why these two device being disabled.
>
> So, can you run kernel with the below debug patch?
>
> diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
> index d45e760..aed319b 100644
> --- a/net/bridge/br_stp_if.c
> +++ b/net/bridge/br_stp_if.c
> @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
> p->state = BR_STATE_DISABLED;
> p->topology_change_ack = 0;
> p->config_pending = 0;
> -
> + dump_stack();
> br_log_state(p);
> br_ifinfo_notify(RTM_NEWLINK, p);
>
> diff --git a/net/core/dev.c b/net/core/dev.c
> index faebb39..9b1617b 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
>
> list_for_each_entry(dev, head, unreg_list) {
> rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
> + dump_stack();
> call_netdevice_notifiers(NETDEV_DOWN, dev);
> }
>
> @@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned int
old_flags)
> if (changes & IFF_UP) {
> if (dev->flags & IFF_UP)
> call_netdevice_notifiers(NETDEV_UP, dev);
> - else
> + else {
> + dump_stack();
> call_netdevice_notifiers(NETDEV_DOWN, dev);
> + }
> }
>
> if (dev->flags & IFF_UP &&
>
>
> Thanks!
>
There you go:
Thank you very much.
[ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
[ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8
[ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8
[ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918
[ 129.084426] Call Trace:
[ 129.084821] virbr0: port 6(veth5) entered disabled state
I can confirm it's the ip command disable the veth device now.
but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of
ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git).
use read /proc/<getppid>/comm to trace which command calls ip.
ip/iplink.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/ip/iplink.c b/ip/iplink.c
index dc98019..ec7ad87 100644
--- a/ip/iplink.c
+++ b/ip/iplink.c
@@ -282,6 +282,7 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
int mtu = -1;
int netns = -1;
int vf = -1;
+ int down = 0;
int numtxqueues = -1;
int numrxqueues = -1;
@@ -293,8 +294,25 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
req->i.ifi_change |= IFF_UP;
req->i.ifi_flags |= IFF_UP;
} else if (strcmp(*argv, "down") == 0) {
+ pid_t ppid = getppid();
+ char file[30];
+ char comm[20];
+ int fd;
+ memset(file, 0, sizeof(file));
+ memset(comm, 0, sizeof(comm));
+ down = 1;
req->i.ifi_change |= IFF_UP;
req->i.ifi_flags &= ~IFF_UP;
+
+ sprintf(file, "/proc/%d/comm", ppid);
+ fd = open(file, O_RDONLY);
+ read(fd, comm, sizeof(comm));
+ comm[strlen(comm) - 1] = '\0';
+ printf("pid %d, comm %s calls ip to down device", ppid, comm);
+ if (*dev)
+ printf(" %s\n", *dev);
+ close(fd);
+
} else if (strcmp(*argv, "name") == 0) {
NEXT_ARG();
*name = *argv;
@@ -475,6 +493,8 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
if (*dev)
duparg2("dev", *argv);
*dev = *argv;
+ if (down)
+ printf(" dev %s\n", *dev);
}
argc--; argv++;
}
--
Thanks!
4:58 a.m.
Am 10.07.2013 11:42, schrieb Gao feng:
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
> Am 10.07.2013 09:03, schrieb Gao feng:
>> On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>>
>>>>
>>>> Yes,actually libvirt did up the veth devices, that's why only
veth2& veth5 are down.
>>>
>>> Where does libvirt up the devices? The debug log does not contain any
"ip link set dev XXX up" commands.
>>> Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
>>>
>>
>> virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
>
> Ahhhh, it's using an ioctl().
>
>>>> I need to know why these two devices are down, I believe they were up,
your bridge and default-net
>>>> looks good. So please show me your kernel message (dmesg), maybe it can
give us some useful information.
>>>
>>> This time veth4 and 5 are down.
>>>
>>> ---cut---
>>
>>> [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
>>> [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
>>> [ 44.473400] virbr0: topology change detected, propagating
>>> [ 44.473407] virbr0: port 5(veth4) entered forwarding state
>>> [ 44.473423] virbr0: port 5(veth4) entered forwarding state
>>
>> veth4 were up here
>>
>>> [ 44.566186] device veth5 entered promiscuous mode
>>> [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
>>> [ 44.571243] virbr0: topology change detected, propagating
>>> [ 44.571250] virbr0: port 6(veth5) entered forwarding state
>>> [ 44.571261] virbr0: port 6(veth5) entered forwarding state
>>> [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
>>> [ 45.000580] virbr0: port 5(veth4) entered disabled state
>>
>> and then it became down.
>>
>>> [ 45.348548] virbr0: port 6(veth5) entered disabled state
>>
>> So, Some places disable the veth4 and veth5.
>> I don't know in which case these two devices will be disabled.
>>
>> I still can't reproduce this problem in my test bed :(
>> I need more information to analyse why these two device being disabled.
>>
>> So, can you run kernel with the below debug patch?
>>
>> diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
>> index d45e760..aed319b 100644
>> --- a/net/bridge/br_stp_if.c
>> +++ b/net/bridge/br_stp_if.c
>> @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
>> p->state = BR_STATE_DISABLED;
>> p->topology_change_ack = 0;
>> p->config_pending = 0;
>> -
>> + dump_stack();
>> br_log_state(p);
>> br_ifinfo_notify(RTM_NEWLINK, p);
>>
>> diff --git a/net/core/dev.c b/net/core/dev.c
>> index faebb39..9b1617b 100644
>> --- a/net/core/dev.c
>> +++ b/net/core/dev.c
>> @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
>>
>> list_for_each_entry(dev, head, unreg_list) {
>> rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
>> + dump_stack();
>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>> }
>>
>> @@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned
int old_flags)
>> if (changes & IFF_UP) {
>> if (dev->flags & IFF_UP)
>> call_netdevice_notifiers(NETDEV_UP, dev);
>> - else
>> + else {
>> + dump_stack();
>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>> + }
>> }
>>
>> if (dev->flags & IFF_UP &&
>>
>>
>> Thanks!
>>
>
> There you go:
>
Thank you very much.
> [ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
> [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8
> [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8
> [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918
> [ 129.084426] Call Trace:
> [ 129.084821] virbr0: port 6(veth5) entered disabled state
>
I can confirm it's the ip command disable the veth device now.
but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of
ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git).
use read /proc/<getppid>/comm to trace which command calls ip.
Will do.
Thanks,
//richard
ip/iplink.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/ip/iplink.c b/ip/iplink.c
index dc98019..ec7ad87 100644
--- a/ip/iplink.c
+++ b/ip/iplink.c
@@ -282,6 +282,7 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
int mtu = -1;
int netns = -1;
int vf = -1;
+ int down = 0;
int numtxqueues = -1;
int numrxqueues = -1;
@@ -293,8 +294,25 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
req->i.ifi_change |= IFF_UP;
req->i.ifi_flags |= IFF_UP;
} else if (strcmp(*argv, "down") == 0) {
+ pid_t ppid = getppid();
+ char file[30];
+ char comm[20];
+ int fd;
+ memset(file, 0, sizeof(file));
+ memset(comm, 0, sizeof(comm));
+ down = 1;
req->i.ifi_change |= IFF_UP;
req->i.ifi_flags &= ~IFF_UP;
+
+ sprintf(file, "/proc/%d/comm", ppid);
+ fd = open(file, O_RDONLY);
+ read(fd, comm, sizeof(comm));
+ comm[strlen(comm) - 1] = '\0';
+ printf("pid %d, comm %s calls ip to down device", ppid, comm);
+ if (*dev)
+ printf(" %s\n", *dev);
+ close(fd);
+
} else if (strcmp(*argv, "name") == 0) {
NEXT_ARG();
*name = *argv;
@@ -475,6 +493,8 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
if (*dev)
duparg2("dev", *argv);
*dev = *argv;
+ if (down)
+ printf(" dev %s\n", *dev);
}
argc--; argv++;
}
2:18 a.m.
Am 10.07.2013 11:42, schrieb Gao feng:
On 07/10/2013 03:23 PM, Richard Weinberger wrote:
> Am 10.07.2013 09:03, schrieb Gao feng:
>> On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>>
>>>>
>>>> Yes,actually libvirt did up the veth devices, that's why only
veth2& veth5 are down.
>>>
>>> Where does libvirt up the devices? The debug log does not contain any
"ip link set dev XXX up" commands.
>>> Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
>>>
>>
>> virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
>
> Ahhhh, it's using an ioctl().
>
>>>> I need to know why these two devices are down, I believe they were up,
your bridge and default-net
>>>> looks good. So please show me your kernel message (dmesg), maybe it can
give us some useful information.
>>>
>>> This time veth4 and 5 are down.
>>>
>>> ---cut---
>>
>>> [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
>>> [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
>>> [ 44.473400] virbr0: topology change detected, propagating
>>> [ 44.473407] virbr0: port 5(veth4) entered forwarding state
>>> [ 44.473423] virbr0: port 5(veth4) entered forwarding state
>>
>> veth4 were up here
>>
>>> [ 44.566186] device veth5 entered promiscuous mode
>>> [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
>>> [ 44.571243] virbr0: topology change detected, propagating
>>> [ 44.571250] virbr0: port 6(veth5) entered forwarding state
>>> [ 44.571261] virbr0: port 6(veth5) entered forwarding state
>>> [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
>>> [ 45.000580] virbr0: port 5(veth4) entered disabled state
>>
>> and then it became down.
>>
>>> [ 45.348548] virbr0: port 6(veth5) entered disabled state
>>
>> So, Some places disable the veth4 and veth5.
>> I don't know in which case these two devices will be disabled.
>>
>> I still can't reproduce this problem in my test bed :(
>> I need more information to analyse why these two device being disabled.
>>
>> So, can you run kernel with the below debug patch?
>>
>> diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
>> index d45e760..aed319b 100644
>> --- a/net/bridge/br_stp_if.c
>> +++ b/net/bridge/br_stp_if.c
>> @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
>> p->state = BR_STATE_DISABLED;
>> p->topology_change_ack = 0;
>> p->config_pending = 0;
>> -
>> + dump_stack();
>> br_log_state(p);
>> br_ifinfo_notify(RTM_NEWLINK, p);
>>
>> diff --git a/net/core/dev.c b/net/core/dev.c
>> index faebb39..9b1617b 100644
>> --- a/net/core/dev.c
>> +++ b/net/core/dev.c
>> @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
>>
>> list_for_each_entry(dev, head, unreg_list) {
>> rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
>> + dump_stack();
>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>> }
>>
>> @@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev, unsigned
int old_flags)
>> if (changes & IFF_UP) {
>> if (dev->flags & IFF_UP)
>> call_netdevice_notifiers(NETDEV_UP, dev);
>> - else
>> + else {
>> + dump_stack();
>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>> + }
>> }
>>
>> if (dev->flags & IFF_UP &&
>>
>>
>> Thanks!
>>
>
> There you go:
>
Thank you very much.
> [ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
> [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6 ffff88003ce7f7b8
> [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103 ffff88003ce7f7e8
> [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000 ffff88003ce7f918
> [ 129.084426] Call Trace:
> [ 129.084821] virbr0: port 6(veth5) entered disabled state
>
I can confirm it's the ip command disable the veth device now.
but I still don't know who calls ip and why.
I search the libvirt code, there are no codes calling "ip link set xxx down".
It's so strange...
Give you an advice, modify the code of
ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git).
use read /proc/<getppid>/comm to trace which command calls ip.
This morning I've installed a wrapper around ip to show me the process tree upon ip
link ... down is used.
The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug
19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
Thanks,
//richard
4:42 a.m.
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
Am 10.07.2013 11:42, schrieb Gao feng:
> On 07/10/2013 03:23 PM, Richard Weinberger wrote:
>> Am 10.07.2013 09:03, schrieb Gao feng:
>>> On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>>>
>>>>>
>>>>> Yes,actually libvirt did up the veth devices, that's why only
veth2& veth5 are down.
>>>>
>>>> Where does libvirt up the devices? The debug log does not contain any
"ip link set dev XXX up" commands.
>>>> Also in src/util/virnetdevveth.c I'm unable to find such a ip
command.
>>>>
>>>
>>> virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
>>
>> Ahhhh, it's using an ioctl().
>>
>>>>> I need to know why these two devices are down, I believe they were
up, your bridge and default-net
>>>>> looks good. So please show me your kernel message (dmesg), maybe it
can give us some useful information.
>>>>
>>>> This time veth4 and 5 are down.
>>>>
>>>> ---cut---
>>>
>>>> [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
>>>> [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
>>>> [ 44.473400] virbr0: topology change detected, propagating
>>>> [ 44.473407] virbr0: port 5(veth4) entered forwarding state
>>>> [ 44.473423] virbr0: port 5(veth4) entered forwarding state
>>>
>>> veth4 were up here
>>>
>>>> [ 44.566186] device veth5 entered promiscuous mode
>>>> [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
>>>> [ 44.571243] virbr0: topology change detected, propagating
>>>> [ 44.571250] virbr0: port 6(veth5) entered forwarding state
>>>> [ 44.571261] virbr0: port 6(veth5) entered forwarding state
>>>> [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
>>>> [ 45.000580] virbr0: port 5(veth4) entered disabled state
>>>
>>> and then it became down.
>>>
>>>> [ 45.348548] virbr0: port 6(veth5) entered disabled state
>>>
>>> So, Some places disable the veth4 and veth5.
>>> I don't know in which case these two devices will be disabled.
>>>
>>> I still can't reproduce this problem in my test bed :(
>>> I need more information to analyse why these two device being disabled.
>>>
>>> So, can you run kernel with the below debug patch?
>>>
>>> diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
>>> index d45e760..aed319b 100644
>>> --- a/net/bridge/br_stp_if.c
>>> +++ b/net/bridge/br_stp_if.c
>>> @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
>>> p->state = BR_STATE_DISABLED;
>>> p->topology_change_ack = 0;
>>> p->config_pending = 0;
>>> -
>>> + dump_stack();
>>> br_log_state(p);
>>> br_ifinfo_notify(RTM_NEWLINK, p);
>>>
>>> diff --git a/net/core/dev.c b/net/core/dev.c
>>> index faebb39..9b1617b 100644
>>> --- a/net/core/dev.c
>>> +++ b/net/core/dev.c
>>> @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
>>>
>>> list_for_each_entry(dev, head, unreg_list) {
>>> rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
>>> + dump_stack();
>>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>>> }
>>>
>>> @@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev,
unsigned int old_flags)
>>> if (changes & IFF_UP) {
>>> if (dev->flags & IFF_UP)
>>> call_netdevice_notifiers(NETDEV_UP, dev);
>>> - else
>>> + else {
>>> + dump_stack();
>>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>>> + }
>>> }
>>>
>>> if (dev->flags & IFF_UP &&
>>>
>>>
>>> Thanks!
>>>
>>
>> There you go:
>>
>
> Thank you very much.
>
>> [ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
>> [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
>> [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6
ffff88003ce7f7b8
>> [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103
ffff88003ce7f7e8
>> [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000
ffff88003ce7f918
>> [ 129.084426] Call Trace:
>
>> [ 129.084821] virbr0: port 6(veth5) entered disabled state
>>
>
> I can confirm it's the ip command disable the veth device now.
> but I still don't know who calls ip and why.
>
> I search the libvirt code, there are no codes calling "ip link set xxx
down".
>
> It's so strange...
>
> Give you an advice, modify the code of
ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git).
> use read /proc/<getppid>/comm to trace which command calls ip.
This morning I've installed a wrapper around ip to show me the process tree upon ip
link ... down is used.
The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug
19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
hmmm...
it's systemd... I have no idea now... :(
4:44 a.m.
Am 11.07.2013 11:42, schrieb Gao feng:
On 07/11/2013 03:18 PM, Richard Weinberger wrote:
> Am 10.07.2013 11:42, schrieb Gao feng:
>> On 07/10/2013 03:23 PM, Richard Weinberger wrote:
>>> Am 10.07.2013 09:03, schrieb Gao feng:
>>>> On 07/10/2013 02:00 PM, Richard Weinberger wrote:
>>>>
>>>>>>
>>>>>> Yes,actually libvirt did up the veth devices, that's why only
veth2& veth5 are down.
>>>>>
>>>>> Where does libvirt up the devices? The debug log does not contain any
"ip link set dev XXX up" commands.
>>>>> Also in src/util/virnetdevveth.c I'm unable to find such a ip
command.
>>>>>
>>>>
>>>> virLXCProcessSetupInterfaceBridged calls virNetDevSetOnline.
>>>
>>> Ahhhh, it's using an ioctl().
>>>
>>>>>> I need to know why these two devices are down, I believe they
were up, your bridge and default-net
>>>>>> looks good. So please show me your kernel message (dmesg), maybe
it can give us some useful information.
>>>>>
>>>>> This time veth4 and 5 are down.
>>>>>
>>>>> ---cut---
>>>>
>>>>> [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
>>>>> [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes
ready
>>>>> [ 44.473400] virbr0: topology change detected, propagating
>>>>> [ 44.473407] virbr0: port 5(veth4) entered forwarding state
>>>>> [ 44.473423] virbr0: port 5(veth4) entered forwarding state
>>>>
>>>> veth4 were up here
>>>>
>>>>> [ 44.566186] device veth5 entered promiscuous mode
>>>>> [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
>>>>> [ 44.571243] virbr0: topology change detected, propagating
>>>>> [ 44.571250] virbr0: port 6(veth5) entered forwarding state
>>>>> [ 44.571261] virbr0: port 6(veth5) entered forwarding state
>>>>> [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes
ready
>>>>> [ 45.000580] virbr0: port 5(veth4) entered disabled state
>>>>
>>>> and then it became down.
>>>>
>>>>> [ 45.348548] virbr0: port 6(veth5) entered disabled state
>>>>
>>>> So, Some places disable the veth4 and veth5.
>>>> I don't know in which case these two devices will be disabled.
>>>>
>>>> I still can't reproduce this problem in my test bed :(
>>>> I need more information to analyse why these two device being disabled.
>>>>
>>>> So, can you run kernel with the below debug patch?
>>>>
>>>> diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
>>>> index d45e760..aed319b 100644
>>>> --- a/net/bridge/br_stp_if.c
>>>> +++ b/net/bridge/br_stp_if.c
>>>> @@ -103,7 +103,7 @@ void br_stp_disable_port(struct net_bridge_port *p)
>>>> p->state = BR_STATE_DISABLED;
>>>> p->topology_change_ack = 0;
>>>> p->config_pending = 0;
>>>> -
>>>> + dump_stack();
>>>> br_log_state(p);
>>>> br_ifinfo_notify(RTM_NEWLINK, p);
>>>>
>>>> diff --git a/net/core/dev.c b/net/core/dev.c
>>>> index faebb39..9b1617b 100644
>>>> --- a/net/core/dev.c
>>>> +++ b/net/core/dev.c
>>>> @@ -1368,6 +1368,7 @@ static int dev_close_many(struct list_head *head)
>>>>
>>>> list_for_each_entry(dev, head, unreg_list) {
>>>> rtmsg_ifinfo(RTM_NEWLINK, dev, IFF_UP|IFF_RUNNING);
>>>> + dump_stack();
>>>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>>>> }
>>>>
>>>> @@ -4729,8 +4730,10 @@ void __dev_notify_flags(struct net_device *dev,
unsigned int old_flags)
>>>> if (changes & IFF_UP) {
>>>> if (dev->flags & IFF_UP)
>>>> call_netdevice_notifiers(NETDEV_UP, dev);
>>>> - else
>>>> + else {
>>>> + dump_stack();
>>>> call_netdevice_notifiers(NETDEV_DOWN, dev);
>>>> + }
>>>> }
>>>>
>>>> if (dev->flags & IFF_UP &&
>>>>
>>>>
>>>> Thanks!
>>>>
>>>
>>> There you go:
>>>
>>
>> Thank you very much.
>>
>>> [ 129.084408] CPU: 1 PID: 4473 Comm: ip Not tainted 3.10.0+ #20
>>> [ 129.084412] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
>>> [ 129.084415] ffff88003760d000 ffff88003ce7f798 ffffffff8172b2a6
ffff88003ce7f7b8
>>> [ 129.084419] ffffffff8154be04 ffff88003760d000 0000000000001103
ffff88003ce7f7e8
>>> [ 129.084422] ffffffff8154be60 0000000000000010 ffff88003760d000
ffff88003ce7f918
>>> [ 129.084426] Call Trace:
>>
>>> [ 129.084821] virbr0: port 6(veth5) entered disabled state
>>>
>>
>> I can confirm it's the ip command disable the veth device now.
>> but I still don't know who calls ip and why.
>>
>> I search the libvirt code, there are no codes calling "ip link set xxx
down".
>>
>> It's so strange...
>>
>> Give you an advice, modify the code of
ip(git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git).
>> use read /proc/<getppid>/comm to trace which command calls ip.
>
> This morning I've installed a wrapper around ip to show me the process tree upon
ip link ... down is used.
> The log showed this:
>
> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug
> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
>
> Now I have to urge to use a "Kantholz". ;-)
>
hmmm...
it's systemd... I have no idea now... :(
TBH it is not systemd's fault.
OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices.
Therefore systemd-udevd called ifup/down and other hotplug magic.
Thanks,
//richard
4:49 a.m.
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
Am 11.07.2013 11:42, schrieb Gao feng:
> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>> This morning I've installed a wrapper around ip to show me the process tree
upon ip link ... down is used.
>> The log showed this:
>>
>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
>> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug
>> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
>>
>> Now I have to urge to use a "Kantholz". ;-)
>>
>
> hmmm...
>
> it's systemd... I have no idea now... :(
TBH it is not systemd's fault.
OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth* devices.
Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse
to fix this ? Can you post a link to the bug here for the sake of archive
records.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
6:58 a.m.
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
> Am 11.07.2013 11:42, schrieb Gao feng:
>> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>>> This morning I've installed a wrapper around ip to show me the process
tree upon ip link ... down is used.
>>> The log showed this:
>>>
>>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
>>> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug
>>> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
>>>
>>> Now I have to urge to use a "Kantholz". ;-)
>>>
>>
>> hmmm...
>>
>> it's systemd... I have no idea now... :(
>
> TBH it is not systemd's fault.
> OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth*
devices.
> Therefore systemd-udevd called ifup/down and other hotplug magic.
Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse
to fix this ? Can you post a link to the bug here for the sake of archive
records.
Sure:
https://bugzilla.novell.com/show_bug.cgi?id=829033
Thanks,
//richard
8:36 p.m.
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
> On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
>> Am 11.07.2013 11:42, schrieb Gao feng:
>>> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>>>> This morning I've installed a wrapper around ip to show me the
process tree upon ip link ... down is used.
>>>> The log showed this:
>>>>
>>>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
>>>> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o
hotplug
>>>> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
>>>>
>>>> Now I have to urge to use a "Kantholz". ;-)
>>>>
>>>
>>> hmmm...
>>>
>>> it's systemd... I have no idea now... :(
>>
>> TBH it is not systemd's fault.
>> OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list veth*
devices.
>> Therefore systemd-udevd called ifup/down and other hotplug magic.
>
> Ah ha, that's a nice issue :-) I assume you've filed a bug against opensuse
> to fix this ? Can you post a link to the bug here for the sake of archive
> records.
Sure:
https://bugzilla.novell.com/show_bug.cgi?id=829033
It's good news we know what causes veth device down. :)
4:33 p.m.
Am 12.07.2013 03:36, schrieb Gao feng:
On 07/11/2013 07:58 PM, Richard Weinberger wrote:
> Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
>> On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
>>> Am 11.07.2013 11:42, schrieb Gao feng:
>>>> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>>>>> This morning I've installed a wrapper around ip to show me the
process tree upon ip link ... down is used.
>>>>> The log showed this:
>>>>>
>>>>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
>>>>> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>>> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>>> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o
hotplug
>>>>> 19910 ? S 0:00 | \_ /sbin/ip link set dev veth5
down
>>>>>
>>>>> Now I have to urge to use a "Kantholz". ;-)
>>>>>
>>>>
>>>> hmmm...
>>>>
>>>> it's systemd... I have no idea now... :(
>>>
>>> TBH it is not systemd's fault.
>>> OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list
veth* devices.
>>> Therefore systemd-udevd called ifup/down and other hotplug magic.
>>
>> Ah ha, that's a nice issue :-) I assume you've filed a bug against
opensuse
>> to fix this ? Can you post a link to the bug here for the sake of archive
>> records.
>
> Sure:
> https://bugzilla.novell.com/show_bug.cgi?id=829033
>
It's good news we know what causes veth device down. :)
How does Fedora deal with veth devices?
SUSE folks think that this is a more likely a libvirt issue and closed my bug report as
invalid...
Thanks,
//richard
4:20 a.m.
On Wed, Jul 17, 2013 at 11:33:22PM +0200, Richard Weinberger wrote:
Am 12.07.2013 03:36, schrieb Gao feng:
> On 07/11/2013 07:58 PM, Richard Weinberger wrote:
>> Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
>>> On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
>>>> Am 11.07.2013 11:42, schrieb Gao feng:
>>>>> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>>>>>> This morning I've installed a wrapper around ip to show me
the process tree upon ip link ... down is used.
>>>>>> The log showed this:
>>>>>>
>>>>>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
>>>>>> 17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>>>> 17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>>>> 17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
>>>>>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5
-o hotplug
>>>>>> 19910 ? S 0:00 | \_ /sbin/ip link set dev
veth5 down
>>>>>>
>>>>>> Now I have to urge to use a "Kantholz". ;-)
>>>>>>
>>>>>
>>>>> hmmm...
>>>>>
>>>>> it's systemd... I have no idea now... :(
>>>>
>>>> TBH it is not systemd's fault.
>>>> OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white list
veth* devices.
>>>> Therefore systemd-udevd called ifup/down and other hotplug magic.
>>>
>>> Ah ha, that's a nice issue :-) I assume you've filed a bug against
opensuse
>>> to fix this ? Can you post a link to the bug here for the sake of archive
>>> records.
>>
>> Sure:
>> https://bugzilla.novell.com/show_bug.cgi?id=829033
>>
>
> It's good news we know what causes veth device down. :)
How does Fedora deal with veth devices?
Well the udev script you mention above does not exist on Fedora and AFAIK
there's no other udev script which runs 'ifconfig down' on NICs.
SUSE folks think that this is a more likely a libvirt issue and
closed my bug report as invalid...
If you remove or modify the 77-network.rules file does it fix the problem.
If so, then it is obviously not a libvirt issue. IMHO it is completely
bogus for udev to be arbitrarily ifdown'ing any interface.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:50 a.m.
Richard,
I think you should have cc'd the bug assignee when discussing this issue
upstream. Adding him now...
Regards,
Jim
Daniel P. Berrange wrote:
On Wed, Jul 17, 2013 at 11:33:22PM +0200, Richard Weinberger wrote:
> Am 12.07.2013 03:36, schrieb Gao feng:
>
>> On 07/11/2013 07:58 PM, Richard Weinberger wrote:
>>
>>> Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
>>>
>>>> On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
>>>>
>>>>> Am 11.07.2013 11:42, schrieb Gao feng:
>>>>>
>>>>>> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>>>>>>
>>>>>>> This morning I've installed a wrapper around ip to show
me the process tree upon ip link ... down is used.
>>>>>>> The log showed this:
>>>>>>>
>>>>>>> 769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
>>>>>>> 17759 ? S 0:00 \_
/usr/lib/systemd/systemd-udevd
>>>>>>> 17764 ? S 0:00 \_
/usr/lib/systemd/systemd-udevd
>>>>>>> 17772 ? S 0:00 \_
/usr/lib/systemd/systemd-udevd
>>>>>>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown
veth5 -o hotplug
>>>>>>> 19910 ? S 0:00 | \_ /sbin/ip link set dev
veth5 down
>>>>>>>
>>>>>>> Now I have to urge to use a "Kantholz". ;-)
>>>>>>>
>>>>>>>
>>>>>> hmmm...
>>>>>>
>>>>>> it's systemd... I have no idea now... :(
>>>>>>
>>>>> TBH it is not systemd's fault.
>>>>> OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not white
list veth* devices.
>>>>> Therefore systemd-udevd called ifup/down and other hotplug magic.
>>>>>
>>>> Ah ha, that's a nice issue :-) I assume you've filed a bug
against opensuse
>>>> to fix this ? Can you post a link to the bug here for the sake of
archive
>>>> records.
>>>>
>>> Sure:
>>> https://bugzilla.novell.com/show_bug.cgi?id=829033
>>>
>>>
>> It's good news we know what causes veth device down. :)
>>
> How does Fedora deal with veth devices?
>
Well the udev script you mention above does not exist on Fedora and AFAIK
there's no other udev script which runs 'ifconfig down' on NICs.
> SUSE folks think that this is a more likely a libvirt issue and closed my bug report
as invalid...
>
If you remove or modify the 77-network.rules file does it fix the problem.
If so, then it is obviously not a libvirt issue. IMHO it is completely
bogus for udev to be arbitrarily ifdown'ing any interface.
Daniel
10:02 a.m.
Am 18.07.2013 16:50, schrieb Jim Fehlig:
Richard,
I think you should have cc'd the bug assignee when discussing this issue
upstream. Adding him now...
Oh, sorry for that!
I thought I did so after pointing Marius to the thread in the mailing list archive,
but obviously I forgot.
Thanks,
//richard
Regards,
Jim
Daniel P. Berrange wrote:
> On Wed, Jul 17, 2013 at 11:33:22PM +0200, Richard Weinberger wrote:
>
>> Am 12.07.2013 03:36, schrieb Gao feng:
>>
>>> On 07/11/2013 07:58 PM, Richard Weinberger wrote:
>>>
>>>> Am 11.07.2013 11:49, schrieb Daniel P. Berrange:
>>>>
>>>>> On Thu, Jul 11, 2013 at 11:44:48AM +0200, Richard Weinberger wrote:
>>>>>
>>>>>> Am 11.07.2013 11:42, schrieb Gao feng:
>>>>>>
>>>>>>> On 07/11/2013 03:18 PM, Richard Weinberger wrote:
>>>>>>>
>>>>>>>> This morning I've installed a wrapper around ip to
show me the process tree upon ip link ... down is used.
>>>>>>>> The log showed this:
>>>>>>>>
>>>>>>>> 769 ? Ss 0:00
/usr/lib/systemd/systemd-udevd
>>>>>>>> 17759 ? S 0:00 \_
/usr/lib/systemd/systemd-udevd
>>>>>>>> 17764 ? S 0:00 \_
/usr/lib/systemd/systemd-udevd
>>>>>>>> 17772 ? S 0:00 \_
/usr/lib/systemd/systemd-udevd
>>>>>>>> 19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown
veth5 -o hotplug
>>>>>>>> 19910 ? S 0:00 | \_ /sbin/ip link set
dev veth5 down
>>>>>>>>
>>>>>>>> Now I have to urge to use a "Kantholz". ;-)
>>>>>>>>
>>>>>>>>
>>>>>>> hmmm...
>>>>>>>
>>>>>>> it's systemd... I have no idea now... :(
>>>>>>>
>>>>>> TBH it is not systemd's fault.
>>>>>> OpenSUSE's /usr/lib/udev/rules.d/77-network.rules did not
white list veth* devices.
>>>>>> Therefore systemd-udevd called ifup/down and other hotplug
magic.
>>>>>>
>>>>> Ah ha, that's a nice issue :-) I assume you've filed a bug
against opensuse
>>>>> to fix this ? Can you post a link to the bug here for the sake of
archive
>>>>> records.
>>>>>
>>>> Sure:
>>>> https://bugzilla.novell.com/show_bug.cgi?id=829033
>>>>
>>>>
>>> It's good news we know what causes veth device down. :)
>>>
>> How does Fedora deal with veth devices?
>>
>
> Well the udev script you mention above does not exist on Fedora and AFAIK
> there's no other udev script which runs 'ifconfig down' on NICs.
>
>
>> SUSE folks think that this is a more likely a libvirt issue and closed my bug
report as invalid...
>>
>
> If you remove or modify the 77-network.rules file does it fix the problem.
> If so, then it is obviously not a libvirt issue. IMHO it is completely
> bogus for udev to be arbitrarily ifdown'ing any interface.
>
> Daniel
>
2:43 p.m.
Am 18.07.2013 17:02, schrieb Richard Weinberger:
Am 18.07.2013 16:50, schrieb Jim Fehlig:
> Richard,
>
> I think you should have cc'd the bug assignee when discussing this issue
> upstream. Adding him now...
Oh, sorry for that!
I thought I did so after pointing Marius to the thread in the mailing list archive,
but obviously I forgot.
Hi!
This is a cleanup call (to stop dhcp-clients if any, remove state, ...)
triggered by the remove event from kernel.
IMO libvirt is missing an "udevadm settle" call between "remove foo; add
foo" -- the "add" and "link up" calls in libvirt are done before
udev finished the "remove" event processing.
Gruesse / Regards,
Marius Tomaschewski <mt(a)suse.de>, <mt(a)suse.com>
--
SUSE LINUX Products GmbH, HRB 16746 (AG Nürnberg),
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer,
Maxfeldstraße 5, 90409 Nürnberg, Germany
3:36 a.m.
On Thu, Jul 18, 2013 at 09:43:54PM +0200, Marius Tomaschewski wrote:
Am 18.07.2013 17:02, schrieb Richard Weinberger:
> Am 18.07.2013 16:50, schrieb Jim Fehlig:
>> Richard,
>>
>> I think you should have cc'd the bug assignee when discussing this issue
>> upstream. Adding him now...
>
> Oh, sorry for that!
> I thought I did so after pointing Marius to the thread in the mailing list archive,
> but obviously I forgot.
Hi!
This is a cleanup call (to stop dhcp-clients if any, remove state, ...)
triggered by the remove event from kernel.
IMO libvirt is missing an "udevadm settle" call between "remove foo; add
foo" -- the "add" and "link up" calls in libvirt are done
before
udev finished the "remove" event processing.
No, these ifup/ifdown udev rules should not be running for veth devices.
They are not normal NICs - they are the same usage scenario as Xen vif
devices or tap devices in QEMU. The suse 77-network.rules script is already
whitelisting the vif* and tap* to avoid this behaviour for those devices
and it should be doing the same for veth* devices too.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
8:41 a.m.
Am 19.07.2013 10:36, schrieb Daniel P. Berrange:
On Thu, Jul 18, 2013 at 09:43:54PM +0200, Marius Tomaschewski wrote:
> Am 18.07.2013 17:02, schrieb Richard Weinberger:
>> Am 18.07.2013 16:50, schrieb Jim Fehlig:
>>> Richard,
>>>
>>> I think you should have cc'd the bug assignee when discussing this issue
>>> upstream. Adding him now...
>>
>> Oh, sorry for that!
>> I thought I did so after pointing Marius to the thread in the mailing list
archive,
>> but obviously I forgot.
>
> Hi!
>
> This is a cleanup call (to stop dhcp-clients if any, remove state, ...)
> triggered by the remove event from kernel.
>
> IMO libvirt is missing an "udevadm settle" call between "remove foo;
add
> foo" -- the "add" and "link up" calls in libvirt are done
before
> udev finished the "remove" event processing.
No, these ifup/ifdown udev rules should not be running for veth devices.
They are not normal NICs - they are the same usage scenario as Xen vif
devices or tap devices in QEMU. The suse 77-network.rules script is already
whitelisting the vif* and tap* to avoid this behaviour for those devices
and it should be doing the same for veth* devices too.
The black list is there as an optimization for interfaces, which ifup
creates itself (e.g. tap). Basically the blacklist can be also removed
and ifup handles its nested calls then (by type, names are free).
We've added vif* as optimization for XEN and when the user wants routed
VM setup using ifup, then the user should specify a name from another
namespace in the VM config.
I can add veth and vnet to the backlist to optimize it once again, but
libvirt needs an udevadm settle call as well or we have same situation
in some time again -- just with a different name.
Without a settle call, I will always consider this as a libvirt issue.
Last but not least, the user can create own udev rules -- there are
customers, where I know that they use own rules.
Gruesse / Regards,
Marius Tomaschewski <mt(a)suse.de>, <mt(a)suse.com>
--
SUSE LINUX Products GmbH, HRB 16746 (AG Nürnberg),
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer,
Maxfeldstraße 5, 90409 Nürnberg, Germany
4:44 a.m.
On Thu, Jul 11, 2013 at 09:18:24AM +0200, Richard Weinberger wrote:
This morning I've installed a wrapper around ip to show me the process tree upon ip
link ... down is used.
The log showed this:
769 ? Ss 0:00 /usr/lib/systemd/systemd-udevd
17759 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
17764 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
17772 ? S 0:00 \_ /usr/lib/systemd/systemd-udevd
19477 ? S 0:00 | \_ /bin/bash /sbin/ifdown veth5 -o hotplug
19910 ? S 0:00 | \_ /sbin/ip link set dev veth5 down
Now I have to urge to use a "Kantholz". ;-)
Ahh, I had a feeling it was going to turn out to be udev's fault.
No I wonder if there is some specific udev rule on your host causing this,
since other people aren't see this behaviour.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4144
days inactive
4162
days old
26 comments
5 participants
participants (5)
-
Daniel P. Berrange -
Gao feng -
Jim Fehlig -
Marius Tomaschewski -
Richard Weinberger