Iptables, et al best practices for protecting KVM host sharing "hostdev" (ixgbe-vf) interfaces with guests
Hi, I'm a developer and I use VM's for testing patches before submitting them (for various linux-* projects, as well as downstream things like OpenWrt, EPEL, etc). I have a trunked 10Gb/s Ethernet connection coming into my KVM servers (all of which are Supermicro x86_servers running CentOS 8) that have igb and ixgbe NIC's, so SR-IOV capable. Here's the situation: * the host is on a secured internal network, called "sandbox", which is VLAN 4. * the "external" network (i.e. public facing on my ISP) is "dirty", and is VLAN 66. * the main subnet that laptops, WAPs, desktops, etc. sit on is "main", and is VLAN 1. The KVM hosts sit wholly inside "sandbox". They host, however, several guests some of which sit on "main", and some of which sit on "main" and "external" (for instance, if I'm testing an OpenWrt firewall configuration with patches and it needs to be publicly accessible so that my testers can peer with it via IPsec). Oh, and one production guest which hosts "git" and sits only on "sandbox". Guests can be CentOS 8S, Windows 11, Ubuntu 20, Debian whatever, Fedora 33 or 34, OpenWrt, etc. My question is this: what's best practices for making sure that a switch VLAN misconfiguration issue, a cabling to the wrong port, etc. doesn't compromise the KVM server itself? How do I allow my KVM server to *not* be on "external", but some of its guests to be, without compromising security? Being the paranoid sort, I'd like to use defense-in-depth so I'm not counting on any single step to protect me. For instance, I might use ebtables or iptables rules to block VLAN traffic on hosts/guests that I don't want to see certain VLAN's under any circumstances. But I might also add interfaces to the list of things that NetworkManager ignores as /etc/NetworkManager/NetworkManager.conf lines like: [keyfile] unmanaged-devices=interface-name:eno*,except:interface-name:eno1 unmanaged-devices=interface-name:enp* Thinking about that, I'm wondering what the notation is for telling NetworkManager to ignore all interfaces except those on VLAN 3, for instance. Would that be: unmanaged-devices=interface-name:eno*.*,except:interface-name:eno*.3 Or something else? And what else can I do to further secure my KVM host? I thought about using one untagged (non VLAN'd) Ethernet for the host, and another tagged Ethernet for the guests, but that just seemed like a false sense of security... Apologies for the cross posting... these questions seems to touch on iptables, libvirt, and NetworkManager all at the same time... Thanks, -Philip
Am 29.10.21 um 02:31 schrieb Philip Prindeville:
Hi,
[…]
Here's the situation:
* the host is on a secured internal network, called "sandbox", which is VLAN 4.
* the "external" network (i.e. public facing on my ISP) is "dirty", and is VLAN 66.
* the main subnet that laptops, WAPs, desktops, etc. sit on is "main", and is VLAN 1.
The KVM hosts sit wholly inside "sandbox".
They host, however, several guests some of which sit on "main", and some of which sit on "main" and "external" (for instance, if I'm testing an OpenWrt firewall configuration with patches and it needs to be publicly accessible so that my testers can peer with it via IPsec). Oh, and one production guest which hosts "git" and sits only on "sandbox".
Guests can be CentOS 8S, Windows 11, Ubuntu 20, Debian whatever, Fedora 33 or 34, OpenWrt, etc.
My question is this: what's best practices for making sure that a switch VLAN misconfiguration issue, a cabling to the wrong port, etc. doesn't compromise the KVM server itself?
How do I allow my KVM server to *not* be on "external", but some of its guests to be, without compromising security?
I have a trunked 10Gb/s Ethernet connection coming into my KVM servers (all of which are Supermicro x86_servers running CentOS 8) that have igb and ixgbe NIC's, so SR-IOV capable. […] Being the paranoid sort, … Just as a side and purely my personal opinion: CentOS seems not the best choice to cure paranoia of the IT type. I also read NetworkManager as your tool of choice. I am concerned with systems favoring an end user targeted abstraction layer over hardened configuration. BSD flavors rely on text files. From this basic layer you stack additional configuration on each layer to connect VMs, workloads and so on being aware of the necessary and possible routes. (I sometimes include dead-man-switches when changing configs so if my ssh session is blocked by misconfiguration it reverts to the previous state after a certain time
How is this supposed to work? How does the server present network connectivity to the guests? If the guests would directly bind to the hardware jumping over the VM server what's the use of the VM then? (If any of the systems in this stack does not bind services to the particular network device it is not reachable through this network. As soon as for example sshd is configured to bind to all network devices it is accessible through any of them available to the VM.) Regarding the switch: think of port duplication on the switch routing all the traffic to a masked hoody wearing laptop owner. How can the guest or host find out? Wrong patching of the cables running virtual LANs (VLAN, multiple networks over the same cable) leaves them intact (from your VMs point of view)? I assume VLAN-configuration is per physical port? Adapting this to VLAN misconfiguration puts your VLAN onto a different port that is not yours? If it is instead of per port a tagging variant and link aggregation is possible than how should you detect – except through bandwidth measurement/ service monitoring (connection reset/ congestion control) – that 4 planned links were reduced to 2? It is very likely I got your scenario wrong. But unless you cannot trace the hops and latency between origin of traffic and endpoint in your VMs you cannot detect configuration changes/ wrong configuration. (If it is the latter no connection at all is very likely.) period.)
My question is this: what's best practices for making sure that a switch VLAN misconfiguration issue, a cabling to the wrong port, etc. doesn't compromise the KVM server itself?
Not sure about best practice. But what about using a macvtap. That by default does not allow host communication and only allows the guests connected to the same master to communicate with each other.
How do I allow my KVM server to *not* be on "external", but some of its guests to be, without compromising security?
Do not configure the interface with an ip address on the host, and make sure you do not have daemons binding to 0.0.0.0 on the host.
participants (3)
-
Benno -
Marc -
Philip Prindeville