1 a.m.
Am 10.07.2013 03:20, schrieb Gao feng:
On 07/09/2013 09:11 PM, Richard Weinberger wrote:
> Am 08.07.2013 05:54, schrieb Gao feng:
>> On 07/05/2013 06:22 PM, Richard Weinberger wrote:
>>> Am 05.07.2013 03:36, schrieb Gao feng:
>>>> On 07/05/2013 04:45 AM, Richard Weinberger wrote:
>>>>> Hi,
>>>>>
>>>>> Am 03.07.2013 12:04, schrieb Gao feng:
>>>>>> Hi,
>>>>>> On 07/01/2013 03:45 PM, Richard Weinberger wrote:
>>>>>>> Hi!
>>>>>>>
>>>>>>> If you have multiple LXC containers with networking and the
autostart feature enabled libvirtd fails to
>>>>>>> up some veth interfaces on the host side.
>>>>>>>
>>>>>>> Most of the time only the first veth device is in state up,
all others are down.
>>>>>>>
>>>>>>> Reproducing is easy.
>>>>>>> 1. Define a few containers (5 in my case)
>>>>>>> 2. Run "virsh autostart ..." on each one.
>>>>>>> 3. stop/start libvirtd
>>>>>>>
>>>>>>> You'll observe that all containers are running, but
"ip a" will report on the host
>>>>>>> side that not all veth devices are up and are not usable
within the containers.
>>>>>>>
>>>>>>> This is not userns related, just retested with libvirt of
today.
>>>>>>
>>>>>> I can not reproduce this problem on my test bed...
>>>>>
>>>>> Strange.
>>>>>
>>>>>> maybe you should wait some seconds for the starting of these
containers.
>>>>>
>>>>> Please see the attached shell script. Using it I'm able to
trigger the issue on all of
>>>>> my test machines.
>>>>> run.sh creates six very minimal containers and enables autostart.
Then it kills and restarts libvirtd.
>>>>> After the script is done you'll see that only one or two veth
devices are up.
>>>>>
>>>>> On the over hand, if I start them manually using a command like this
one:
>>>>> for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start
test-$cfg ; done
>>>>> All veths are always up.
>>>>>
>>>>
>>>>
>>>> I still can not reproduce even use your script.
>>>>
>>>> [root@Donkey-I5 Desktop]# ./run.sh
>>>> Domain test-a defined from container_a.conf
>>>>
>>>> Domain test-a marked as autostarted
>>>>
>>>> Domain test-b defined from container_b.conf
>>>>
>>>> Domain test-b marked as autostarted
>>>>
>>>> Domain test-c defined from container_c.conf
>>>>
>>>> Domain test-c marked as autostarted
>>>>
>>>> Domain test-d defined from container_d.conf
>>>>
>>>> Domain test-d marked as autostarted
>>>>
>>>> Domain test-e defined from container_e.conf
>>>>
>>>> Domain test-e marked as autostarted
>>>>
>>>> Domain test-f defined from container_f.conf
>>>>
>>>> Domain test-f marked as autostarted
>>>>
>>>> 2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0
>>>> 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 :
outputs=1:file:/home/gaofeng/libvirtd.log
>>>> waiting a bit....
>>>> 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>> 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master virbr0 state UP qlen 1000
>>>>
>>>>
>>>> Can you post your libvirt debug log?
>>>
>>> Please see attached file.
>>>
>>> 43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master
virbr0 state DOWN qlen 1000
>>> 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UP qlen 1000
>>> 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master
virbr0 state DOWN qlen 100
>>>
>>
>> strange, I can not see veth related error message from your log.
>>
>> seems like all of the veth devices of host had been up but for some reasons they
become down.
>
> I think libvirt has to do "ip link set dev vethX up".
> Otherwise the device state is undefined.
>
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are
down.
Where does libvirt up the devices? The debug log does not contain any "ip link set
dev XXX up" commands.
Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
I need to know why these two devices are down, I believe they were
up, your bridge and default-net
looks good. So please show me your kernel message (dmesg), maybe it can give us some
useful information.
This time veth4 and 5 are down.
---cut---
[ 4.167119] systemd[1]: Starting File System Check on Root Device...
[ 5.150740] mount (725) used greatest stack depth: 4616 bytes left
[ 5.397652] systemd-udevd[755]: starting version 195
[ 6.155648] EXT4-fs (vda2): re-mounted. Opts: acl,user_xattr
[ 6.369755] systemd-journald[730]: Received SIGUSR1
[ 6.877588] Adding 690172k swap on /dev/vda1. Priority:-1 extents:1 across:690172k
[ 8.205295] ip (1446) used greatest stack depth: 4584 bytes left
[ 9.397258] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 9.397275] 8021q: adding VLAN 0 to HW filter on device eth0
[ 9.397733] ip (1690) used greatest stack depth: 4456 bytes left
[ 11.399834] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 11.401711] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 41.179748] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready
[ 42.126303] cgroup: libvirtd (2736) created nested cgroup for controller
"memory" which has incomplete hierarchy support. Nested cgroups may change
behavior in the future.
[ 42.126307] cgroup: "memory" requires setting use_hierarchy to 1 on the
root.
[ 42.126599] cgroup: libvirtd (2736) created nested cgroup for controller
"blkio" which has incomplete hierarchy support. Nested cgroups may change
behavior in the future.
[ 42.145511] ip (2843) used greatest stack depth: 4136 bytes left
[ 42.149231] device veth0 entered promiscuous mode
[ 42.152167] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[ 42.559264] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[ 42.559336] virbr0: topology change detected, propagating
[ 42.559343] virbr0: port 1(veth0) entered forwarding state
[ 42.559359] virbr0: port 1(veth0) entered forwarding state
[ 42.559398] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready
[ 42.702130] ip (3070) used greatest stack depth: 4088 bytes left
[ 42.708164] device veth1 entered promiscuous mode
[ 42.712190] IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready
[ 42.712197] virbr0: topology change detected, propagating
[ 42.712202] virbr0: port 2(veth1) entered forwarding state
[ 42.712212] virbr0: port 2(veth1) entered forwarding state
[ 43.147155] virbr0: port 2(veth1) entered disabled state
[ 43.225250] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
[ 43.225322] virbr0: topology change detected, propagating
[ 43.225328] virbr0: port 2(veth1) entered forwarding state
[ 43.225358] virbr0: port 2(veth1) entered forwarding state
[ 43.325556] ip (3293) used greatest stack depth: 4064 bytes left
[ 43.330133] device veth2 entered promiscuous mode
[ 43.334201] IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready
[ 43.334208] virbr0: topology change detected, propagating
[ 43.334214] virbr0: port 3(veth2) entered forwarding state
[ 43.334224] virbr0: port 3(veth2) entered forwarding state
[ 43.613184] IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready
[ 43.767050] ip (3521) used greatest stack depth: 3992 bytes left
[ 43.773832] device veth3 entered promiscuous mode
[ 43.778169] IPv6: ADDRCONF(NETDEV_UP): veth3: link is not ready
[ 43.778177] virbr0: topology change detected, propagating
[ 43.778182] virbr0: port 4(veth3) entered forwarding state
[ 43.778193] virbr0: port 4(veth3) entered forwarding state
[ 44.076299] IPv6: ADDRCONF(NETDEV_CHANGE): veth3: link becomes ready
[ 44.153214] device veth4 entered promiscuous mode
[ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready
[ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready
[ 44.473400] virbr0: topology change detected, propagating
[ 44.473407] virbr0: port 5(veth4) entered forwarding state
[ 44.473423] virbr0: port 5(veth4) entered forwarding state
[ 44.566186] device veth5 entered promiscuous mode
[ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready
[ 44.571243] virbr0: topology change detected, propagating
[ 44.571250] virbr0: port 6(veth5) entered forwarding state
[ 44.571261] virbr0: port 6(veth5) entered forwarding state
[ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready
[ 45.000580] virbr0: port 5(veth4) entered disabled state
[ 45.348548] virbr0: port 6(veth5) entered disabled state
[ 45.348837] ip (4349) used greatest stack depth: 3944 bytes left
---cut---
Thanks,
//richard