Am 10.07.2013 03:20, schrieb Gao feng:
On 07/09/2013 09:11 PM, Richard Weinberger wrote:
Am 08.07.2013 05:54, schrieb Gao feng:
On 07/05/2013 06:22 PM, Richard Weinberger wrote:
Am 05.07.2013 03:36, schrieb Gao feng:
On 07/05/2013 04:45 AM, Richard Weinberger wrote:
Hi,
Am 03.07.2013 12:04, schrieb Gao feng: > Hi, > On 07/01/2013 03:45 PM, Richard Weinberger wrote: >> Hi! >> >> If you have multiple LXC containers with networking and the autostart feature enabled libvirtd fails to >> up some veth interfaces on the host side. >> >> Most of the time only the first veth device is in state up, all others are down. >> >> Reproducing is easy. >> 1. Define a few containers (5 in my case) >> 2. Run "virsh autostart ..." on each one. >> 3. stop/start libvirtd >> >> You'll observe that all containers are running, but "ip a" will report on the host >> side that not all veth devices are up and are not usable within the containers. >> >> This is not userns related, just retested with libvirt of today. > > I can not reproduce this problem on my test bed...
Strange.
> maybe you should wait some seconds for the starting of these containers.
Please see the attached shell script. Using it I'm able to trigger the issue on all of my test machines. run.sh creates six very minimal containers and enables autostart. Then it kills and restarts libvirtd. After the script is done you'll see that only one or two veth devices are up.
On the over hand, if I start them manually using a command like this one: for cfg in a b c d e f ; do /opt/libvirt/bin/virsh -c lxc:/// start test-$cfg ; done All veths are always up.
I still can not reproduce even use your script.
[root@Donkey-I5 Desktop]# ./run.sh Domain test-a defined from container_a.conf
Domain test-a marked as autostarted
Domain test-b defined from container_b.conf
Domain test-b marked as autostarted
Domain test-c defined from container_c.conf
Domain test-c marked as autostarted
Domain test-d defined from container_d.conf
Domain test-d marked as autostarted
Domain test-e defined from container_e.conf
Domain test-e marked as autostarted
Domain test-f defined from container_f.conf
Domain test-f marked as autostarted
2013-07-05 01:26:47.155+0000: 27163: info : libvirt version: 1.1.0 2013-07-05 01:26:47.155+0000: 27163: debug : virLogParseOutputs:1334 : outputs=1:file:/home/gaofeng/libvirtd.log waiting a bit.... 167: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 169: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 171: veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 173: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 175: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 177: veth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000
Can you post your libvirt debug log?
Please see attached file.
43: veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 45: veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 47: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000 49: veth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 51: veth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UP qlen 1000 53: veth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 100
strange, I can not see veth related error message from your log.
seems like all of the veth devices of host had been up but for some reasons they become down.
I think libvirt has to do "ip link set dev vethX up". Otherwise the device state is undefined.
Yes,actually libvirt did up the veth devices, that's why only veth2& veth5 are down.
Where does libvirt up the devices? The debug log does not contain any "ip link set dev XXX up" commands. Also in src/util/virnetdevveth.c I'm unable to find such a ip command.
I need to know why these two devices are down, I believe they were up, your bridge and default-net looks good. So please show me your kernel message (dmesg), maybe it can give us some useful information.
This time veth4 and 5 are down. ---cut--- [ 4.167119] systemd[1]: Starting File System Check on Root Device... [ 5.150740] mount (725) used greatest stack depth: 4616 bytes left [ 5.397652] systemd-udevd[755]: starting version 195 [ 6.155648] EXT4-fs (vda2): re-mounted. Opts: acl,user_xattr [ 6.369755] systemd-journald[730]: Received SIGUSR1 [ 6.877588] Adding 690172k swap on /dev/vda1. Priority:-1 extents:1 across:690172k [ 8.205295] ip (1446) used greatest stack depth: 4584 bytes left [ 9.397258] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 9.397275] 8021q: adding VLAN 0 to HW filter on device eth0 [ 9.397733] ip (1690) used greatest stack depth: 4456 bytes left [ 11.399834] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 11.401711] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 41.179748] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready [ 42.126303] cgroup: libvirtd (2736) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 42.126307] cgroup: "memory" requires setting use_hierarchy to 1 on the root. [ 42.126599] cgroup: libvirtd (2736) created nested cgroup for controller "blkio" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 42.145511] ip (2843) used greatest stack depth: 4136 bytes left [ 42.149231] device veth0 entered promiscuous mode [ 42.152167] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready [ 42.559264] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready [ 42.559336] virbr0: topology change detected, propagating [ 42.559343] virbr0: port 1(veth0) entered forwarding state [ 42.559359] virbr0: port 1(veth0) entered forwarding state [ 42.559398] IPv6: ADDRCONF(NETDEV_CHANGE): virbr0: link becomes ready [ 42.702130] ip (3070) used greatest stack depth: 4088 bytes left [ 42.708164] device veth1 entered promiscuous mode [ 42.712190] IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready [ 42.712197] virbr0: topology change detected, propagating [ 42.712202] virbr0: port 2(veth1) entered forwarding state [ 42.712212] virbr0: port 2(veth1) entered forwarding state [ 43.147155] virbr0: port 2(veth1) entered disabled state [ 43.225250] IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready [ 43.225322] virbr0: topology change detected, propagating [ 43.225328] virbr0: port 2(veth1) entered forwarding state [ 43.225358] virbr0: port 2(veth1) entered forwarding state [ 43.325556] ip (3293) used greatest stack depth: 4064 bytes left [ 43.330133] device veth2 entered promiscuous mode [ 43.334201] IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready [ 43.334208] virbr0: topology change detected, propagating [ 43.334214] virbr0: port 3(veth2) entered forwarding state [ 43.334224] virbr0: port 3(veth2) entered forwarding state [ 43.613184] IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready [ 43.767050] ip (3521) used greatest stack depth: 3992 bytes left [ 43.773832] device veth3 entered promiscuous mode [ 43.778169] IPv6: ADDRCONF(NETDEV_UP): veth3: link is not ready [ 43.778177] virbr0: topology change detected, propagating [ 43.778182] virbr0: port 4(veth3) entered forwarding state [ 43.778193] virbr0: port 4(veth3) entered forwarding state [ 44.076299] IPv6: ADDRCONF(NETDEV_CHANGE): veth3: link becomes ready [ 44.153214] device veth4 entered promiscuous mode [ 44.158209] IPv6: ADDRCONF(NETDEV_UP): veth4: link is not ready [ 44.473317] IPv6: ADDRCONF(NETDEV_CHANGE): veth4: link becomes ready [ 44.473400] virbr0: topology change detected, propagating [ 44.473407] virbr0: port 5(veth4) entered forwarding state [ 44.473423] virbr0: port 5(veth4) entered forwarding state [ 44.566186] device veth5 entered promiscuous mode [ 44.571234] IPv6: ADDRCONF(NETDEV_UP): veth5: link is not ready [ 44.571243] virbr0: topology change detected, propagating [ 44.571250] virbr0: port 6(veth5) entered forwarding state [ 44.571261] virbr0: port 6(veth5) entered forwarding state [ 44.902308] IPv6: ADDRCONF(NETDEV_CHANGE): veth5: link becomes ready [ 45.000580] virbr0: port 5(veth4) entered disabled state [ 45.348548] virbr0: port 6(veth5) entered disabled state [ 45.348837] ip (4349) used greatest stack depth: 3944 bytes left ---cut--- Thanks, //richard