[libvirt-users] Guests can't connect to each other
Hi, I'm using libvirt and qemu on Debian Wheezy. I'm having a strange behavior. Guests can't connect to each other when they're on the same host. On the host I'm using bonding (in active / backup mode) and vlan. It looks like this : eth0 \ / macvtap0 bond0 --- vlan222 eth1 / \ macvtap1 So I've got two guests, let's say A and B. When I try to ping B from A, it works : # ping -s 3000 -c 5 78.109.95.11 PING 78.109.95.11 (78.109.95.11) 3000(3028) bytes of data. 3008 bytes from 78.109.95.11: icmp_req=1 ttl=64 time=0.065 ms 3008 bytes from 78.109.95.11: icmp_req=2 ttl=64 time=2.19 ms 3008 bytes from 78.109.95.11: icmp_req=3 ttl=64 time=1.43 ms --- 78.109.95.11 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.065/0.960/2.197/0.760 ms But nothing happens when I try to ssh it (not even a timeout). You'll find enclosed the tcpdump captures on the source and the destination. It's the same when I use netcat in udp. At the same time, connection from the host to one guest is working perfectly. There is no iptables rule on the host, and nothing too on the guests. Here are the virsh dumpxml of the different components : # virsh dumpxml vm1 <domain type='kvm' id='11'> <name>vm1</name> <uuid>4eaaed00-c610-b468-ad55-600a0b4e244c</uuid> <memory>2048000</memory> <currentMemory>2048000</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu cpuset='0,2,4,8,10,12,14'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0,8'/> <vcpupin vcpu='1' cpuset='2,10'/> <vcpupin vcpu='2' cpuset='4,12'/> <vcpupin vcpu='3' cpuset='6,14'/> <vcpupin vcpu='4' cpuset='0,8'/> <vcpupin vcpu='5' cpuset='2,10'/> <vcpupin vcpu='6' cpuset='4,12'/> <vcpupin vcpu='7' cpuset='6,14'/> </cputune> <os> <type arch='x86_64' machine='pc-1.0'>hvm</type> <boot dev='hd'/> <boot dev='network'/> </os> <features> <acpi/> <apic/> </features> <cpu match='exact'> <model>Westmere</model> <vendor>Intel</vendor> <topology sockets='1' cores='8' threads='1'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='rdtscp'/> <feature policy='require' name='ht'/> <feature policy='require' name='dca'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='pclmuldq'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> <feature policy='require' name='monitor'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/vps/vm1'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <interface type='network'> <mac address='52:54:00:0e:58:ae'/> <source network='vlan222'/> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='fr'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='vga' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> </domain> # virsh dumpxml vm2 <domain type='kvm' id='13'> <name>vm2</name> <uuid>4f760831-22b1-ff3b-26e7-6b3fec49e918</uuid> <memory>2048000</memory> <currentMemory>2048000</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu cpuset='1,3,5,7,9,11,13,15'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='1' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='2' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='3' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='4' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='5' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='6' cpuset='1,3,5,7,9,11,13,15'/> <vcpupin vcpu='7' cpuset='1,3,5,7,9,11,13,15'/> </cputune> <os> <type arch='x86_64' machine='pc-1.0'>hvm</type> <boot dev='hd'/> <boot dev='network'/> </os> <features> <acpi/> <apic/> </features> <cpu match='exact'> <model>Westmere</model> <vendor>Intel</vendor> <topology sockets='1' cores='4' threads='2'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='rdtscp'/> <feature policy='require' name='ht'/> <feature policy='require' name='dca'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='pclmuldq'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> <feature policy='require' name='monitor'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/vps/vm2'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <interface type='network'> <mac address='52:54:00:cb:ce:41'/> <source network='vlan222'/> <target dev='macvtap1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0' keymap='fr'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='vga' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> </domain> # virsh net-dumpxml vlan222 <network> <name>vlan222</name> <uuid>2b763b5c-4ec1-9b5f-b29d-b7a7ea0f743d</uuid> <forward dev='vlan222' mode='bridge'> <interface dev='vlan222'/> </forward> </network> Thanks in advance to help me understand this issue.
On Fri, Apr 13, 2012 at 02:33:24PM +0200, Anthony Bourguignon wrote:
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange behavior. Guests can't connect to each other when they're on the same host.
On the host I'm using bonding (in active / backup mode) and vlan. It looks like this : eth0 \ / macvtap0 bond0 --- vlan222 eth1 / \ macvtap1
I thought it was a known problem that macvtap guests can't talk to each other ...? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On 04/13/2012 03:17 PM, Richard W.M. Jones wrote:
On Fri, Apr 13, 2012 at 02:33:24PM +0200, Anthony Bourguignon wrote:
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange behavior. Guests can't connect to each other when they're on the same host.
On the host I'm using bonding (in active / backup mode) and vlan. It looks like this : eth0 \ / macvtap0 bond0 --- vlan222 eth1 / \ macvtap1 I thought it was a known problem that macvtap guests can't talk to each other ...?
guests can't talk directly to the *host*. They can talk to each other though, as long as "bridge" mode is uses. In "private", "vepa" and "passthrough" modes, *all* traffic goes only out to the switch, so any communication between guests must be via the switch (and if the switch doesn't support that, they can't talk).
On 04/13/2012 08:33 AM, Anthony Bourguignon wrote:
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange behavior. Guests can't connect to each other when they're on the same host.
On the host I'm using bonding (in active / backup mode) and vlan. It looks like this : eth0 \ / macvtap0 bond0 --- vlan222 eth1 / \ macvtap1
So I've got two guests, let's say A and B. When I try to ping B from A, it works : # ping -s 3000 -c 5 78.109.95.11 PING 78.109.95.11 (78.109.95.11) 3000(3028) bytes of data. 3008 bytes from 78.109.95.11: icmp_req=1 ttl=64 time=0.065 ms 3008 bytes from 78.109.95.11: icmp_req=2 ttl=64 time=2.19 ms 3008 bytes from 78.109.95.11: icmp_req=3 ttl=64 time=1.43 ms
--- 78.109.95.11 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.065/0.960/2.197/0.760 ms
But nothing happens when I try to ssh it (not even a timeout). You'll find enclosed the tcpdump captures on the source and the destination. It's the same when I use netcat in udp.
Your config looks fine (the important part is that you're using bridge mode for macvatap rather than private). I would suspect some sort of bug related to using macvtap on a vlan device (or, even more, a vlan connected to a bond). Try changing your network config to use 1) a vlan connected directly to eth0 or eth1, rather than the bond, 2) bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give you an idea of which part of the equation isn't working.
Le dimanche 15 avril 2012 à 10:19 -0400, Laine Stump a écrit :
Your config looks fine (the important part is that you're using bridge mode for macvatap rather than private). I would suspect some sort of bug related to using macvtap on a vlan device (or, even more, a vlan connected to a bond). Try changing your network config to use 1) a vlan connected directly to eth0 or eth1, rather than the bond, 2) bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give you an idea of which part of the equation isn't working.
I've just tested with a vlan directly connected to eth0 (1) and eth0 directly (3) and I've still got the same behavior. That's really strange. I will test with bridge interfaces later. If you've got other ideas, please let me know. Thanks
On 04/16/2012 07:02 AM, Anthony Bourguignon wrote:
Your config looks fine (the important part is that you're using bridge mode for macvatap rather than private). I would suspect some sort of bug related to using macvtap on a vlan device (or, even more, a vlan connected to a bond). Try changing your network config to use 1) a vlan connected directly to eth0 or eth1, rather than the bond, 2) bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give you an idea of which part of the equation isn't working. I've just tested with a vlan directly connected to eth0 (1) and eth0
Le dimanche 15 avril 2012 à 10:19 -0400, Laine Stump a écrit : directly (3) and I've still got the same behavior. That's really strange.
So if you define a network like the following, and connect the guests to it, the guests can all communicate with the outside world, but not with each other: <network> <name>eth0-direct</name> <forward dev='eth0' mode='bridge'> <interface dev='eth0'/> </forward> </network> This would lead me to believe a problem in your kernel's macvtap "bridge" mode - it really sounds like it's insisting on using "private" mode instead. I'm not conversant on what macvtap bugs were fixed in what kernel version, but you could start by looking to see if there's a later version you could easily upgrade to.
I just noticed the packet dumps in your original message. They show that the TCP checksum of all packets past #3 are incorrect. This is a similar problem with one caused by using the virtio "vhost" mode with standard tap network devices - if the dhcp server and client are on the same physical machine, the sender of a packet doesn't fill in the TCP checksum (assuming that tcp checksum offloading will fix it), but the tcp checksum offload happens at a level low enough it is never reached by packets just going to another guest on the same host, so the checksum is never filled in, and the receiver thus rejects the packet. (If you want to read a long boring history of this problem, which is only tangentially related to yours, look here: https://bugzilla.redhat.com/show_bug.cgi?id=612588 ) It's possible there is a similar bug with certain versions of macvtap support in the kernel; you'd have to look that up. In the meantime, I think you should be able to solve your problem by modifying your guests' interface definitions like this: <interface type='network'> <mac address='52:54:00:0e:58:ae'/> <source network='vlan222'/> <target dev='macvtap0'/> <model type='virtio'/> <driver name='qemu'/> <!-- ADD THIS LINE --> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> This forces libvirt to call qemu with vhost turned off.
Hello, Le lundi 16 avril 2012 à 14:38 -0400, Laine Stump a écrit :
I just noticed the packet dumps in your original message. They show that the TCP checksum of all packets past #3 are incorrect.
This is a similar problem with one caused by using the virtio "vhost" mode with standard tap network devices - if the dhcp server and client are on the same physical machine, the sender of a packet doesn't fill in the TCP checksum (assuming that tcp checksum offloading will fix it), but the tcp checksum offload happens at a level low enough it is never reached by packets just going to another guest on the same host, so the checksum is never filled in, and the receiver thus rejects the packet.
For information, I'm using kernel 3.2.14-1 from Debian Wheezy with qemu-kvm package 1.0+dfsg-11 . Quite recent versions so. The bug are probably not present anymore. I've tried to add the line <driver name='qemu'/> in the xml description but I've still got the same behavior. The first packets are ok but the following ones does have a wrong checksum. Here is the command line executed by libvirt : /usr/bin/kvm -S -M pc-1.0 -cpu core2duo,+lahf_lm,+rdtscp,+pdpe1gb,+aes, +popcnt,+sse4.2,+sse4.1,+dca,+pdcm,+xtpr,+cx16,+tm2,+est,+smx,+vmx, +ds_cpl,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds -enable-kvm -m 2000 -mem-prealloc -mem-path /mnt/hugepages/libvirt/qemu -smp 8,sockets=1,cores=8,threads=1 -name vm1,process=qemu:vm1 -uuid 4eaaed00-c610-b468-ad55-600a0b4e244c -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/vps/vm1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0e:58:ae,bus=pci.0,addr=0x3,bootindex=2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 0.0.0.0:0 -k fr -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 Apparently, vhost_net is not activated (I've unloaded the module to be sure and the kernel process is not showing). I'm also wondering about this output : # ip link show macvtap0 27: macvtap0@vlan222: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500 link/ether 52:54:00:0e:58:ae brd ff:ff:ff:ff:ff:ff Why mode is DEFAULT ? Shouldn't it be bridge ?
participants (3)
-
Anthony Bourguignon -
Laine Stump -
Richard W.M. Jones