[libvirt-users] macvtap and tagged VLANs to the VM
Hi, I would like to run a network firewall as a VM on a KVM host. There are ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0. To save myself from configuring all VLANs on the KVM host, I'd like to hand the entire ethernet link to the VM and to have the VLAN interfaces there. Using classical Linux bridges (brctl), things work fine. They don't when I try macvlan: On the host: 4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 5: unt382@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 15: macvtap3@enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 500 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 macvtap mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff inet6 fe80::20d:b9ff:fe34:2afe/64 scope link valid_lft forever preferred_lft forever 5: unt382@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff inet6 fe80::20d:b9ff:fe34:2afe/64 scope link valid_lft forever preferred_lft forever 15: macvtap3@enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever In the XML: <interface type='direct'> <mac address='52:54:00:bf:bb:ab'/> <source dev='enp3s0' mode='bridge'/> <target dev='macvtap3'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> And in the VM: root@grml ~ # ip -d link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 3: vlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 root@grml ~ # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever 3: vlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet 192.168.252.220/24 brd 192.168.252.255 scope global vlan0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever root@grml ~ # I then ping from the VM to 192.168.252.241, which is a differnt host on the network, neither the VM host the VM is running on nor another VM on the same host. That should rule out the connectivity issues that a macvtap interface has, right? On the VM, I see ARP requests going out, but no answers come in. On the pinged host, I see: 22:50:23.881163 52:54:00:bf:bb:ab > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.252.241 tell 192.168.252.220, length 46 22:50:23.881242 52:54:00:95:df:a6 > 52:54:00:bf:bb:ab, ethertype ARP (0x0806), length 42: Reply 192.168.252.241 is-at 52:54:00:95:df:a6, length 28 So, the packets going out from my VM are correctly delivered to the target, the target replies, but the replies never make it back to the VM. Do I see correctly that tcpdump on the VM host won't give accurate readings since macvtap will divert the frame before tcpdump will see it? On the other hand, a VM directly configured to the host's unt382 interface works fine: <interface type='direct'> <mac address='52:54:00:cb:ed:34'/> <source dev='unt382' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> I would however like to avoid having 25 interface stanzas in my XML. I would appeciate any ideas to solve this issue. I know this is most probably not a libvirt issue, but this list is about the only place that comes to my mind where people knowledgeable about those complex network stuff might hang around. If there is a better place to ask, I am open for suggestion. Please pardon my intrusion. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Hi, nobody? If this is the wrong forum, where can I find people who can help with this issue? Greetings Marc On Sun, Dec 16, 2018 at 10:59:22PM +0100, Marc Haber wrote:
From: Marc Haber <mh+libvirt-users@zugschlus.de> Subject: macvtap and tagged VLANs to the VM To: libvirt-users@redhat.com Date: Sun, 16 Dec 2018 22:59:22 +0100 User-Agent: Mutt/1.9.5 (2018-04-13)
Hi,
I would like to run a network firewall as a VM on a KVM host. There are ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.
To save myself from configuring all VLANs on the KVM host, I'd like to hand the entire ethernet link to the VM and to have the VLAN interfaces there. Using classical Linux bridges (brctl), things work fine.
They don't when I try macvlan:
On the host: 4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 5: unt382@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 15: macvtap3@enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 500 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 macvtap mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff inet6 fe80::20d:b9ff:fe34:2afe/64 scope link valid_lft forever preferred_lft forever 5: unt382@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff inet6 fe80::20d:b9ff:fe34:2afe/64 scope link valid_lft forever preferred_lft forever 15: macvtap3@enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever
In the XML: <interface type='direct'> <mac address='52:54:00:bf:bb:ab'/> <source dev='enp3s0' mode='bridge'/> <target dev='macvtap3'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
And in the VM: root@grml ~ # ip -d link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 3: vlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 root@grml ~ # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever 3: vlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet 192.168.252.220/24 brd 192.168.252.255 scope global vlan0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever root@grml ~ #
I then ping from the VM to 192.168.252.241, which is a differnt host on the network, neither the VM host the VM is running on nor another VM on the same host. That should rule out the connectivity issues that a macvtap interface has, right? On the VM, I see ARP requests going out, but no answers come in.
On the pinged host, I see: 22:50:23.881163 52:54:00:bf:bb:ab > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.252.241 tell 192.168.252.220, length 46 22:50:23.881242 52:54:00:95:df:a6 > 52:54:00:bf:bb:ab, ethertype ARP (0x0806), length 42: Reply 192.168.252.241 is-at 52:54:00:95:df:a6, length 28
So, the packets going out from my VM are correctly delivered to the target, the target replies, but the replies never make it back to the VM.
Do I see correctly that tcpdump on the VM host won't give accurate readings since macvtap will divert the frame before tcpdump will see it?
On the other hand, a VM directly configured to the host's unt382 interface works fine: <interface type='direct'> <mac address='52:54:00:cb:ed:34'/> <source dev='unt382' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> I would however like to avoid having 25 interface stanzas in my XML.
I would appeciate any ideas to solve this issue. I know this is most probably not a libvirt issue, but this list is about the only place that comes to my mind where people knowledgeable about those complex network stuff might hang around. If there is a better place to ask, I am open for suggestion. Please pardon my intrusion.
Greetings Marc
-- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
-- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
On 12/16/18 4:59 PM, Marc Haber wrote:
Hi,
I would like to run a network firewall as a VM on a KVM host. There are ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.
To save myself from configuring all VLANs on the KVM host, I'd like to hand the entire ethernet link to the VM and to have the VLAN interfaces there. Using classical Linux bridges (brctl), things work fine.
When I asked the person I go to with questions about macvtap (because he knows the internals), his response was "if a Linux host bridge works, then he should use that". In other words, he was skeptical that what you want to do could be made to work with macvtap. Is there a specific reason you need to use macvtap than a Linux host bridge?
They don't when I try macvlan:
On the host: 4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 5: unt382@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 15: macvtap3@enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 500 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 macvtap mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff inet6 fe80::20d:b9ff:fe34:2afe/64 scope link valid_lft forever preferred_lft forever 5: unt382@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff inet6 fe80::20d:b9ff:fe34:2afe/64 scope link valid_lft forever preferred_lft forever 15: macvtap3@enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever
In the XML: <interface type='direct'> <mac address='52:54:00:bf:bb:ab'/> <source dev='enp3s0' mode='bridge'/> <target dev='macvtap3'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
And in the VM: root@grml ~ # ip -d link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 3: vlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 root@grml ~ # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever 3: vlan0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff inet 192.168.252.220/24 brd 192.168.252.255 scope global vlan0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:febf:bbab/64 scope link valid_lft forever preferred_lft forever root@grml ~ #
I then ping from the VM to 192.168.252.241, which is a differnt host on the network, neither the VM host the VM is running on nor another VM on the same host. That should rule out the connectivity issues that a macvtap interface has, right? On the VM, I see ARP requests going out, but no answers come in.
On the pinged host, I see: 22:50:23.881163 52:54:00:bf:bb:ab > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.252.241 tell 192.168.252.220, length 46 22:50:23.881242 52:54:00:95:df:a6 > 52:54:00:bf:bb:ab, ethertype ARP (0x0806), length 42: Reply 192.168.252.241 is-at 52:54:00:95:df:a6, length 28
So, the packets going out from my VM are correctly delivered to the target, the target replies, but the replies never make it back to the VM.
Do I see correctly that tcpdump on the VM host won't give accurate readings since macvtap will divert the frame before tcpdump will see it?
On the other hand, a VM directly configured to the host's unt382 interface works fine: <interface type='direct'> <mac address='52:54:00:cb:ed:34'/> <source dev='unt382' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> I would however like to avoid having 25 interface stanzas in my XML.
I would appeciate any ideas to solve this issue. I know this is most probably not a libvirt issue, but this list is about the only place that comes to my mind where people knowledgeable about those complex network stuff might hang around. If there is a better place to ask, I am open for suggestion. Please pardon my intrusion.
Greetings Marc
Hi Laine, thanks for your answer, I really appreciate that. On Wed, Jan 02, 2019 at 11:34:30AM -0500, Laine Stump wrote:
On 12/16/18 4:59 PM, Marc Haber wrote:
I would like to run a network firewall as a VM on a KVM host. There are ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.
To save myself from configuring all VLANs on the KVM host, I'd like to hand the entire ethernet link to the VM and to have the VLAN interfaces there. Using classical Linux bridges (brctl), things work fine.
When I asked the person I go to with questions about macvtap (because he knows the internals), his response was "if a Linux host bridge works, then he should use that". In other words, he was skeptical that what you want to do could be made to work with macvtap.
I see. A Linux host bridge is what I build with brctl?
Is there a specific reason you need to use macvtap than a Linux host bridge?
I somehow got the impression that using macvtap is the more "modern" and also more performant approach to bring network to VMs. Since the VM in question is a Firewall, I'd love to have the performance impact caused by virtualization minimized[1]. If this is a misconception, it might have been partially caused by some colleagues at my last customer's site who very vocal about deprecating the classical brctl bridges in favor of macvtap/macvlan, and the fact that virt-manager uses macvtap by default and needs to be massaged into allowing a classic brctl bridge. Greetings Marc [1] The transfer rate of a tunneled IPv6 link with a dedicated VM handling the tunnel and a dedicated VM handling firewalling with brctl bridges (ingress packet - hypervisor - firewall VM - hypervisor - tunnel VM - hypervisor - firewall VM - hypervisor - egress packet) maxes out at about 15 Mbit on the APU device being used, with negligible load on the two VMs and the hypervisor kernel spending a non-negligible amount of its time inside the kernel wich I interpret as the context changes killing the machine -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
On 1/3/19 9:23 AM, Marc Haber wrote:
Hi Laine,
thanks for your answer, I really appreciate that.
On Wed, Jan 02, 2019 at 11:34:30AM -0500, Laine Stump wrote:
On 12/16/18 4:59 PM, Marc Haber wrote:
I would like to run a network firewall as a VM on a KVM host. There are ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.
To save myself from configuring all VLANs on the KVM host, I'd like to hand the entire ethernet link to the VM and to have the VLAN interfaces there. Using classical Linux bridges (brctl), things work fine.
When I asked the person I go to with questions about macvtap (because he knows the internals), his response was "if a Linux host bridge works, then he should use that". In other words, he was skeptical that what you want to do could be made to work with macvtap.
I see.
A Linux host bridge is what I build with brctl?
Yes, although I wouldn't use the brctl command directly if I were you - much simpler an more stable to set it up in your host's system network config, and let initscripts/NetworkManager/whatever your distro uses take care of creating the bridge / attaching your physical ethernet each time the host boots.
Is there a specific reason you need to use macvtap than a Linux host bridge?
I somehow got the impression that using macvtap is the more "modern" and also more performant approach to bring network to VMs. Since the VM in question is a Firewall, I'd love to have the performance impact caused by virtualization minimized[1].
If this is a misconception, it might have been partially caused by some colleagues at my last customer's site who very vocal about deprecating the classical brctl bridges in favor of macvtap/macvlan,
Not really. macvtap is useful because it's simple to setup (doesn't require a change to the host's network config), but is also problematic in some ways, e.g. it doesn't allow host<->guest communication (unless you have a switch that reflects all traffic back to the sender). A long time ago people were theorizing that macvtap would provide much better performance than a host bridge connection, but I don't think that has been the case in practice (it may be somewhat better in some situations, but not really in others). These days use of macvtap for guest connection is more the exception than the rule.
and the fact that virt-manager uses macvtap by default and needs to be massaged into allowing a classic brctl bridge.
On my host that has a bridge named "br0", *that* is what is offered for connection of a new guest interface by default, not any macvtap interface. And on another host that has no br0 device, libvirt's own "default" network is the default selection for new network devices, followed in the list by other libvirt virtual networks. In all cases, the selections for macvtap connections to the host's physical ethernets are included all the way at the bottom of the list, below both libvirt virtual networks *and* host bridges.
Greetings Marc
[1] The transfer rate of a tunneled IPv6 link with a dedicated VM handling the tunnel and a dedicated VM handling firewalling with brctl bridges (ingress packet - hypervisor - firewall VM - hypervisor - tunnel VM - hypervisor - firewall VM - hypervisor - egress packet) maxes out at about 15 Mbit on the APU device being used,
1) I guess this APU is something other than x86? 15Mbits is *glacially* slow, regardless of what's used for the connection. 2) have you tried the same setup with macvtap (since you can't get vlans working, maybe just try an apples-apples comparison of traffic with no vlans on both setups) and seen markedly better performance? 3) You should be able to get some amount of improvement with host bridges if you define a libvirt network pointing at the bridge you're using, with macTableManager="libvirt" in the definition. e.g. if you have a bridge in your host network config called "br0", define this network: <network> <name>nolearn</name> <bridge name='br0'macTableManager='libvirt'/> <forward mode='bridge'/> </network> then use this for your guest interfaces: <interface type='network'/> <source network='nolearn'/> ... </interface> Setting macTableManager='libvirt' causes libvirt to disable learning on the bridge ports, and manually update the FDB of the bridge with the MAC addresses of the guest interfaces. If learning is disabled on all ports except one (that "one" being the physical ethernet of the host that is attached to the bridge), then the kernel also disables promiscuous mode on the physical ethernet. The combination of disabling learning and disabling promiscuous mode on the physdev should have a noticeable impact on performance (although I'm not certain if this panned out as nicely in practice as it did in the minds of kernel network developers either :-) (note that if you play around with this you'll need to understand that only traffic sent to the broadcast MAC, and to the specific MAC of the guest as specified in the libvirt config for the guest, will actually make it to the guest - no playing tricks with modifying the MAC address only in the guest!)
with negligible load on the two VMs and the hypervisor kernel spending a non-negligible amount of its time inside the kernel wich I interpret as the context changes killing the machine
On Thu, Jan 03, 2019 at 05:15:29PM -0500, Laine Stump wrote:
On 1/3/19 9:23 AM, Marc Haber wrote:
On Wed, Jan 02, 2019 at 11:34:30AM -0500, Laine Stump wrote:
On 12/16/18 4:59 PM, Marc Haber wrote:
I would like to run a network firewall as a VM on a KVM host. There are ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.
To save myself from configuring all VLANs on the KVM host, I'd like to hand the entire ethernet link to the VM and to have the VLAN interfaces there. Using classical Linux bridges (brctl), things work fine.
When I asked the person I go to with questions about macvtap (because he knows the internals), his response was "if a Linux host bridge works, then he should use that". In other words, he was skeptical that what you want to do could be made to work with macvtap.
I see.
A Linux host bridge is what I build with brctl?
Yes, although I wouldn't use the brctl command directly if I were you - much simpler an more stable to set it up in your host's system network config, and let initscripts/NetworkManager/whatever your distro uses take care of creating the bridge / attaching your physical ethernet each time the host boots.
Of course. Network is managed with systemd-networkd on the system in question. I was just trying to make clear which construct I mean.
Is there a specific reason you need to use macvtap than a Linux host bridge?
I somehow got the impression that using macvtap is the more "modern" and also more performant approach to bring network to VMs. Since the VM in question is a Firewall, I'd love to have the performance impact caused by virtualization minimized[1].
If this is a misconception, it might have been partially caused by some colleagues at my last customer's site who very vocal about deprecating the classical brctl bridges in favor of macvtap/macvlan,
Not really. macvtap is useful because it's simple to setup (doesn't require a change to the host's network config), but is also problematic in some
Ah. If that's the only reason to use it, then it it of couse unsuitable for the more complex setups.
ways, e.g. it doesn't allow host<->guest communication (unless you have a switch that reflects all traffic back to the sender).
I have my hosts on a different VLAN than the guests, so that traffic between host and guest is forced over the router and the firewall. That is, IMO, the most standard-conforming way to solve the issue.
These days use of macvtap for guest connection is more the exception than the rule.
Interesting, that was not my perception. Thanks for setting me straight on that.
and the fact that virt-manager uses macvtap by default and needs to be massaged into allowing a classic brctl bridge.
On my host that has a bridge named "br0", *that* is what is offered for connection of a new guest interface by default, not any macvtap interface.
Interesting. I had to create a dummy network in libvirt. But I guess that you have generated that network from inside libvirt, right? That does not work for me since I use systemd-networkd, and Debian's netcf version cannot interface with systemd-networkd. Also I'd like my network to be up withlit libvirt ;-)
And on another host that has no br0 device, libvirt's own "default" network is the default selection for new network devices, followed in the list by other libvirt virtual networks. In all cases, the selections for macvtap connections to the host's physical ethernets are included all the way at the bottom of the list, below both libvirt virtual networks *and* host bridges.
It might be my own fault that my libvirt needs to be massaged into showing the actually preferred interface. I'll use brctl bridges in my setup. Thanks for setting me straight. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
participants (2)
-
Laine Stump -
Marc Haber