7:33 a.m.
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange
behavior. Guests can't connect to each other when they're on the same
host.
On the host I'm using bonding (in active / backup mode) and vlan. It
looks like this :
eth0 \ / macvtap0
bond0 --- vlan222
eth1 / \ macvtap1
So I've got two guests, let's say A and B. When I try to ping B from A,
it works :
# ping -s 3000 -c 5 78.109.95.11
PING 78.109.95.11 (78.109.95.11) 3000(3028) bytes of data.
3008 bytes from 78.109.95.11: icmp_req=1 ttl=64 time=0.065 ms
3008 bytes from 78.109.95.11: icmp_req=2 ttl=64 time=2.19 ms
3008 bytes from 78.109.95.11: icmp_req=3 ttl=64 time=1.43 ms
--- 78.109.95.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.065/0.960/2.197/0.760 ms
But nothing happens when I try to ssh it (not even a timeout). You'll
find enclosed the tcpdump captures on the source and the destination.
It's the same when I use netcat in udp.
At the same time, connection from the host to one guest is working
perfectly.
There is no iptables rule on the host, and nothing too on the guests.
Here are the virsh dumpxml of the different components :
# virsh dumpxml vm1
<domain type='kvm' id='11'>
<name>vm1</name>
<uuid>4eaaed00-c610-b468-ad55-600a0b4e244c</uuid>
<memory>2048000</memory>
<currentMemory>2048000</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu cpuset='0,2,4,8,10,12,14'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0,8'/>
<vcpupin vcpu='1' cpuset='2,10'/>
<vcpupin vcpu='2' cpuset='4,12'/>
<vcpupin vcpu='3' cpuset='6,14'/>
<vcpupin vcpu='4' cpuset='0,8'/>
<vcpupin vcpu='5' cpuset='2,10'/>
<vcpupin vcpu='6' cpuset='4,12'/>
<vcpupin vcpu='7' cpuset='6,14'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-1.0'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu match='exact'>
<model>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='8' threads='1'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='est'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='rdtscp'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='pclmuldq'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='monitor'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'
io='native'/>
<source dev='/dev/vps/vm1'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>
<interface type='network'>
<mac address='52:54:00:0e:58:ae'/>
<source network='vlan222'/>
<target dev='macvtap0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/0'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/0'>
<source path='/dev/pts/0'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes'
listen='0.0.0.0'
keymap='fr'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='vga' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
# virsh dumpxml vm2
<domain type='kvm' id='13'>
<name>vm2</name>
<uuid>4f760831-22b1-ff3b-26e7-6b3fec49e918</uuid>
<memory>2048000</memory>
<currentMemory>2048000</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu cpuset='1,3,5,7,9,11,13,15'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='1' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='2' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='3' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='4' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='5' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='6' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='7' cpuset='1,3,5,7,9,11,13,15'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-1.0'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu match='exact'>
<model>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='4' threads='2'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='est'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='rdtscp'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='pclmuldq'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='monitor'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'
io='native'/>
<source dev='/dev/vps/vm2'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>
<interface type='network'>
<mac address='52:54:00:cb:ce:41'/>
<source network='vlan222'/>
<target dev='macvtap1'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5901' autoport='yes'
listen='0.0.0.0'
keymap='fr'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='vga' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
# virsh net-dumpxml vlan222
<network>
<name>vlan222</name>
<uuid>2b763b5c-4ec1-9b5f-b29d-b7a7ea0f743d</uuid>
<forward dev='vlan222' mode='bridge'>
<interface dev='vlan222'/>
</forward>
</network>
Thanks in advance to help me understand this issue.
2:17 p.m.
On Fri, Apr 13, 2012 at 02:33:24PM +0200, Anthony Bourguignon wrote:
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange
behavior. Guests can't connect to each other when they're on the same
host.
On the host I'm using bonding (in active / backup mode) and vlan. It
looks like this :
eth0 \ / macvtap0
bond0 --- vlan222
eth1 / \ macvtap1
I thought it was a known problem that macvtap guests can't
talk to each other ...?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
9:21 a.m.
On 04/13/2012 03:17 PM, Richard W.M. Jones wrote:
On Fri, Apr 13, 2012 at 02:33:24PM +0200, Anthony Bourguignon wrote:
> Hi,
>
> I'm using libvirt and qemu on Debian Wheezy. I'm having a strange
> behavior. Guests can't connect to each other when they're on the same
> host.
>
> On the host I'm using bonding (in active / backup mode) and vlan. It
> looks like this :
> eth0 \ / macvtap0
> bond0 --- vlan222
> eth1 / \ macvtap1
I thought it was a known problem that macvtap guests can't
talk to each other ...?
guests can't talk directly to the *host*. They can talk to each other
though, as long as "bridge" mode is uses. In "private",
"vepa" and
"passthrough" modes, *all* traffic goes only out to the switch, so any
communication between guests must be via the switch (and if the switch
doesn't support that, they can't talk).
9:19 a.m.
On 04/13/2012 08:33 AM, Anthony Bourguignon wrote:
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange
behavior. Guests can't connect to each other when they're on the same
host.
On the host I'm using bonding (in active / backup mode) and vlan. It
looks like this :
eth0 \ / macvtap0
bond0 --- vlan222
eth1 / \ macvtap1
So I've got two guests, let's say A and B. When I try to ping B from A,
it works :
# ping -s 3000 -c 5 78.109.95.11
PING 78.109.95.11 (78.109.95.11) 3000(3028) bytes of data.
3008 bytes from 78.109.95.11: icmp_req=1 ttl=64 time=0.065 ms
3008 bytes from 78.109.95.11: icmp_req=2 ttl=64 time=2.19 ms
3008 bytes from 78.109.95.11: icmp_req=3 ttl=64 time=1.43 ms
--- 78.109.95.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.065/0.960/2.197/0.760 ms
But nothing happens when I try to ssh it (not even a timeout). You'll
find enclosed the tcpdump captures on the source and the destination.
It's the same when I use netcat in udp.
Your config looks fine (the important part is that you're using bridge
mode for macvatap rather than private). I would suspect some sort of bug
related to using macvtap on a vlan device (or, even more, a vlan
connected to a bond). Try changing your network config to use
1) a vlan connected directly to eth0 or eth1, rather than the bond, 2)
bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give
you an idea of which part of the equation isn't working.
6:02 a.m.
Le dimanche 15 avril 2012 à 10:19 -0400, Laine Stump a écrit :
Your config looks fine (the important part is that you're using
bridge
mode for macvatap rather than private). I would suspect some sort of bug
related to using macvtap on a vlan device (or, even more, a vlan
connected to a bond). Try changing your network config to use
1) a vlan connected directly to eth0 or eth1, rather than the bond, 2)
bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give
you an idea of which part of the equation isn't working.
I've just tested with a vlan directly connected to eth0 (1) and eth0
directly (3) and I've still got the same behavior. That's really
strange.
I will test with bridge interfaces later. If you've got other ideas,
please let me know.
Thanks
1:13 p.m.
On 04/16/2012 07:02 AM, Anthony Bourguignon wrote:
Le dimanche 15 avril 2012 à 10:19 -0400, Laine Stump a écrit :
> Your config looks fine (the important part is that you're using bridge
> mode for macvatap rather than private). I would suspect some sort of bug
> related to using macvtap on a vlan device (or, even more, a vlan
> connected to a bond). Try changing your network config to use
> 1) a vlan connected directly to eth0 or eth1, rather than the bond, 2)
> bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give
> you an idea of which part of the equation isn't working.
I've just tested with a vlan directly connected to eth0 (1) and eth0
directly (3) and I've still got the same behavior. That's really
strange.
So if you define a network like the following, and connect the guests to
it, the guests can all communicate with the outside world, but not with
each other:
<network>
<name>eth0-direct</name>
<forward dev='eth0' mode='bridge'>
<interface dev='eth0'/>
</forward>
</network>
This would lead me to believe a problem in your kernel's macvtap
"bridge" mode - it really sounds like it's insisting on using
"private"
mode instead. I'm not conversant on what macvtap bugs were fixed in what
kernel version, but you could start by looking to see if there's a later
version you could easily upgrade to.
1:38 p.m.
I just noticed the packet dumps in your original message. They show that
the TCP checksum of all packets past #3 are incorrect.
This is a similar problem with one caused by using the virtio "vhost"
mode with standard tap network devices - if the dhcp server and client
are on the same physical machine, the sender of a packet doesn't fill
in the TCP checksum (assuming that tcp checksum offloading will fix it),
but the tcp checksum offload happens at a level low enough it is never
reached by packets just going to another guest on the same host, so the
checksum is never filled in, and the receiver thus rejects the packet.
(If you want to read a long boring history of this problem, which is
only tangentially related to yours, look here:
https://bugzilla.redhat.com/show_bug.cgi?id=612588 )
It's possible there is a similar bug with certain versions of macvtap
support in the kernel; you'd have to look that up. In the meantime, I
think you should be able to solve your problem by modifying your guests'
interface definitions like this:
<interface type='network'>
<mac address='52:54:00:0e:58:ae'/>
<source network='vlan222'/>
<target dev='macvtap0'/>
<model type='virtio'/>
<driver name='qemu'/> <!-- ADD THIS LINE -->
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
This forces libvirt to call qemu with vhost turned off.
4:46 a.m.
Hello,
Le lundi 16 avril 2012 à 14:38 -0400, Laine Stump a écrit :
I just noticed the packet dumps in your original message. They show
that
the TCP checksum of all packets past #3 are incorrect.
This is a similar problem with one caused by using the virtio "vhost"
mode with standard tap network devices - if the dhcp server and client
are on the same physical machine, the sender of a packet doesn't fill
in the TCP checksum (assuming that tcp checksum offloading will fix it),
but the tcp checksum offload happens at a level low enough it is never
reached by packets just going to another guest on the same host, so the
checksum is never filled in, and the receiver thus rejects the packet.
For information, I'm using kernel 3.2.14-1 from Debian Wheezy with
qemu-kvm package 1.0+dfsg-11 . Quite recent versions so. The bug are
probably not present anymore.
I've tried to add the line <driver name='qemu'/> in the xml description
but I've still got the same behavior. The first packets are ok but the
following ones does have a wrong checksum.
Here is the command line executed by libvirt :
/usr/bin/kvm -S -M pc-1.0 -cpu core2duo,+lahf_lm,+rdtscp,+pdpe1gb,+aes,
+popcnt,+sse4.2,+sse4.1,+dca,+pdcm,+xtpr,+cx16,+tm2,+est,+smx,+vmx,
+ds_cpl,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds -enable-kvm -m 2000
-mem-prealloc -mem-path /mnt/hugepages/libvirt/qemu -smp
8,sockets=1,cores=8,threads=1 -name vm1,process=qemu:vm1 -uuid
4eaaed00-c610-b468-ad55-600a0b4e244c -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive
file=/dev/vps/vm1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=21,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0e:58:ae,bus=pci.0,addr=0x3,bootindex=2
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
0.0.0.0:0 -k fr -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Apparently, vhost_net is not activated (I've unloaded the module to be
sure and the kernel process is not showing).
I'm also wondering about this output :
# ip link show macvtap0
27: macvtap0@vlan222: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether 52:54:00:0e:58:ae brd ff:ff:ff:ff:ff:ff
Why mode is DEFAULT ? Shouldn't it be bridge ?
4482
days inactive
4487
days old
7 comments
3 participants
participants (3)
-
Anthony Bourguignon -
Laine Stump -
Richard W.M. Jones