I have three hosts running Ubuntu 12.04 (libvirt 0.9.8). The configuration is one host running on bare metal while the other two are KVM guests. The first guest is my network router. It has a direct connection to a physical nic going out to the internet, and a bridged connection to a nic for the lan. The host has ip forwarding enabled and forwards my lan traffic back and forth to the internet. The second host is a "lan" machine, which is also on the bridged lan nic. There are several other physical hosts also on the switched lan network this nic connects to. Both hosts are configured on the lan tap as follows (different mac): <interface type='direct'> <mac address='13:54:21:1f:f3:42'/> <source dev='eth1' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> Everything on the firewall host seems to work OK. Hosts elsewhere on the LAN can connect to it, and they have no problem routing through it and out to the internet and back. That is, traffic passes through both nics and this host successfully. The other guest can also be reached successfully from machines on the LAN. I can ping it and I can ssh to it. I also note that it seems to be able to be able to talk to the "router" host over the vtap bridge: it can perform dns lookups against the router host, and they can seem to reach each other's ports. Where I'm getting stuck is that for whatever reason, the second guest apparently cannot reach the internet via my router host. It's the only host anywhere on the lan that apparently can't pass forwarding traffic via the router guest, and the only common feature appears to be the macvtap bridge. This issue *does not* happen when using a common linux bridge in the otherwise same configuration and the same hosts. In the linux bridge scenario, the lan guest forwards traffic via the router guest fine. But when switching to the macvtap configuration, suddenly the lan guest no longer forwards traffic via the router guest and out to the internet. What appears to be happening is that the traffic is crossing the bridge, but the router host does not classify it or masq it properly, and it never makes the internet trip as expected. Further extending my suspicions, initiating an outbound http connection does not raise an entry in the conntrack table on the router for the problem host. So my question is, what is it about macvtap bridge that would cause traffic coming over via the tap bridge to be routed differently than traffic coming *up* the bridge from the physical interface, or via a traditional linux bridge?

I'm attaching a basic diagram for the topology I'm having the issue with. Guest 1 and Guest 2 are both KVM guests. Guest 1 is the network router with interfaces to the LAN and WAN on the host. Guest 2 is just a regular KVM guest with no routing/forwarding; it shares the LAN interface on the host. When using macvtap bridge, all connections are routable *except* for the green dotted line. In this scenario, traffic coming from Guest 2 reaches Guest 1, but forwarding fails to happen. The traffic never goes out the WAN interface, and isn't being dropped by the firewall. It's as if forwarding is off (except it isn't). Traffic coming from the physical interface and up the tap never has this problem. It's only traffic crossing the tap bridge.

If linux bridge is substituted for macvtap, traffic over the green dotted line then routes correctly and Guest 2 can reach WAN via Guest 1. No other changes are required for this to work.

On Mon, Apr 8, 2013 at 3:24 PM, Laine Stump <laine(a)laine.org&gt; wrote: > On 04/04/2013 10:48 PM, Matt LaPlante wrote: >> I have three hosts running Ubuntu 12.04 (libvirt 0.9.8). The >> configuration is one host running on bare metal while the other two >> are KVM guests. >> >> The first guest is my network router. It has a direct connection to a >> physical nic going out to the internet, and a bridged connection to a >> nic for the lan. The host has ip forwarding enabled and forwards my >> lan traffic back and forth to the internet. >> >> The second host is a "lan" machine, which is also on the bridged lan >> nic. There are several other physical hosts also on the switched lan >> network this nic connects to. >> >> Both hosts are configured on the lan tap as follows (different mac): >> >> <interface type='direct'> >> <mac address='13:54:21:1f:f3:42'/> >> <source dev='eth1' mode='bridge'/> >> <model type='virtio'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> >> </interface> >> >> Everything on the firewall host seems to work OK. Hosts elsewhere on >> the LAN can connect to it, and they have no problem routing through it >> and out to the internet and back. That is, traffic passes through >> both nics and this host successfully. >> >> The other guest can also be reached successfully from machines on the >> LAN. I can ping it and I can ssh to it. I also note that it seems to >> be able to be able to talk to the "router" host over the vtap bridge: >> it can perform dns lookups against the router host, and they can seem >> to reach each other's ports. >> >> Where I'm getting stuck is that for whatever reason, the second guest >> apparently cannot reach the internet via my router host. It's the >> only host anywhere on the lan that apparently can't pass forwarding >> traffic via the router guest, and the only common feature appears to >> be the macvtap bridge. This issue *does not* happen when using a >> common linux bridge in the otherwise same configuration and the same >> hosts. In the linux bridge scenario, the lan guest forwards traffic >> via the router guest fine. But when switching to the macvtap >> configuration, suddenly the lan guest no longer forwards traffic via >> the router guest and out to the internet. >> >> What appears to be happening is that the traffic is crossing the >> bridge, but the router host does not classify it or masq it properly, >> and it never makes the internet trip as expected. Further extending >> my suspicions, initiating an outbound http connection does not raise >> an entry in the conntrack table on the router for the problem host. >> >> So my question is, what is it about macvtap bridge that would cause >> traffic coming over via the tap bridge to be routed differently than >> traffic coming *up* the bridge from the physical interface, or via a >> traditional linux bridge? > The macvtap data path is architected differently than, for example, the > data path for a tap + linux host bridge. One big difference is that, > even in macvtap "bridge" mode, you can't use that interface to > communicate between a guest and its host - packets from the guest are > injected into the packet stream at a point too low to have the > possibility of reflecting them back up the stack to the host's IP. > > Because it seems like you've mixed use of the words "host" and "guest" > in your description, I was unable to follow exactly what the setup is > and which paths don't function, but when a host bridge setup works and a > macvtap setup doesn't, the point that I said above is almost always the > reason. > > There's a description of the problem and one possible workaround here: > > http://wiki.libvirt.org/page/Guest_can_reach_outside_network%2C_but_can%2... > > If that's not the problem, when you reply please give more concrete > details about your topology, and be careful to use "host" only when > talking about the virtualization host, and always use "guest" when > talking about a virtual machine. > > > > _______________________________________________ > libvirt-users mailing list > libvirt-users(a)redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users