How can I control iptables/nftables rules addition on libvirtd host on Debian 12 ?
Hello, When I install libvirt-daemon on a Debian 12 host, I can see the iptables rules below beeing added. *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :LIBVIRT_FWI - [0:0] :LIBVIRT_FWO - [0:0] :LIBVIRT_FWX - [0:0] :LIBVIRT_INP - [0:0] :LIBVIRT_OUT - [0:0] -A INPUT -j LIBVIRT_INP -A FORWARD -j LIBVIRT_FWX -A FORWARD -j LIBVIRT_FWI -A FORWARD -j LIBVIRT_FWO -A OUTPUT -j LIBVIRT_OUT -A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT COMMIT For some reason, I need to add a couple of other rules. How can I do that ? Best regards
On 1/29/25 8:39 AM, oza.4h07@gmail.com wrote:
Hello,
When I install libvirt-daemon on a Debian 12 host, I can see the iptables rules below beeing added.
*filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :LIBVIRT_FWI - [0:0] :LIBVIRT_FWO - [0:0] :LIBVIRT_FWX - [0:0] :LIBVIRT_INP - [0:0] :LIBVIRT_OUT - [0:0] -A INPUT -j LIBVIRT_INP -A FORWARD -j LIBVIRT_FWX -A FORWARD -j LIBVIRT_FWI -A FORWARD -j LIBVIRT_FWO -A OUTPUT -j LIBVIRT_OUT -A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT COMMIT
For some reason, I need to add a couple of other rules. How can I do that ?
I assume you are actually using libvirt's "default" virtual network, and that the rules you want to add are related to that network? If the rules are unrelated, you can set them up however / wherever you like. But it these rule are directly related to the virtual network created by libvirt, then take a look at https://libvirt.org/hooks.html You can add extra rules by creating the file /etc/libvirt/hooks/network (which is a shell script) and running the appropriate nftables/iptables commands there. The above linked page will show you how to construct the script so that commands to add rules (or other stuff) are run when the network starts, and other commands to remove/undo the rules/stuff are run when the network is stopped. (BTW, if your distro has libvirt 10.4.0 or newer, you can tell it to use nftables rules rather than iptables - just add: firewall_backend = "nftables" to /etc/libvirt/network.conf)
On Wed, Jan 29, 2025 at 12:24:47PM -0500, Laine Stump wrote:
On 1/29/25 8:39 AM, oza.4h07@gmail.com wrote: (BTW, if your distro has libvirt 10.4.0 or newer, you can tell it to use nftables rules rather than iptables - just add:
firewall_backend = "nftables"
to /etc/libvirt/network.conf)
Debian 12 doesn't come with a new enough libvirt version anyway, but FYI a few months back I switched the default backend in Debian to nftables (matching Fedora) only to walk back the decision after getting several reports of it breaking software that's just too popular to ignore. See [1] for more details. I don't expect that Debian will be able to move off the iptables backend any time soon, at least when it comes to the default. Changing the backend on a per-system basis is of course totally possible, as long as you understand the caveats. [1] https://bugs.debian.org/1090355 -- Andrea Bolognani / Red Hat / Virtualization
On 1/30/25 10:48 AM, Andrea Bolognani wrote:
On Wed, Jan 29, 2025 at 12:24:47PM -0500, Laine Stump wrote:
On 1/29/25 8:39 AM, oza.4h07@gmail.com wrote: (BTW, if your distro has libvirt 10.4.0 or newer, you can tell it to use nftables rules rather than iptables - just add:
firewall_backend = "nftables"
to /etc/libvirt/network.conf)
Debian 12 doesn't come with a new enough libvirt version anyway, but FYI a few months back I switched the default backend in Debian to nftables (matching Fedora) only to walk back the decision after getting several reports of it breaking software that's just too popular to ignore. See [1] for more details.
I don't expect that Debian will be able to move off the iptables backend any time soon, at least when it comes to the default. Changing the backend on a per-system basis is of course totally possible, as long as you understand the caveats.
Sigh. In the days of iptables, there were 3 main tables (filter, nat, and mangle) and everybody's rules went into those same 3 tables. Within that single table, if a packet reached a REJECT or DROP rule before an ACCEPT rule (or the end of the table) then the packet would be dropped, but if it reached an ACCEPT rule first, then it would never see the REJECT rule, and be accepted. But with nftables, there are an infinite number of "base tables", all traffic is evaluated against *all* tables *all the way* to either accept/reject in *all* cases, and it must get to the end of *every single table* without encountering a reject rule in order for the traffic to be accepted - there is no "early exit on accept" that skips all the rest of the tables if the traffic is accepted by one table. (yeah, I know there's a lot of words enclosed in *..* there) So the reason that traffic is flowing with libvirt's iptables backend + docker/whatever is that libvirt loads its rules *last* and so it can override the "other guy's" "REJECT iptables rulesby inserting its own ACCEPT rules at the beginning of the chain - the docket/whatever rules are never even encountered. But when libvirt uses the nftables backend, it creates its own base level table (just as firewalld does). If docker/whatever is still using iptables, its iptables rules are converted into nftables rules and added to a table named "filter". So now each packet is processed through the libvirt table up until it reaches a resolution, and then it's processed through the entire filter table until it reaches a resolution - if either of the tables leads to a reject result, then the traffic is dropped. The only way around this is to add mirrored ACCEPT rules to the "filter" table (ie where all the iptables-converted-to-nftables rules are located) that match the traffic libvirt wants to accept. If we do this, then we're just using iptables again, which is what we're trying to *get away from*. And this isn't a temporary issue caused by docker/whatever remining on iptables - once they modernize and begin using nftables, it will be the same situation, except instead of the "other" table being "filter", it will be a different table created just for docker/whatever (e.g. called "docker") and we'll be back at the same problem. (Note that this exact same problem will occur if, for example, someone installs docker on a system with firewalld or ufw or whatever. If you don't see problems in these cases, then its because one of the two packages is adding in extra rules to the other package's table to accept the traffic they want accepted) There is no generic way to fix this problem. libvirt can't possible find every possible firewall system and add rules to the table of every single one that passes traffic from libvirt guests. I guess the best we can theoretically do is make a list of "supported firewall enemies" and add in extra stuff just like we currently do with firewalld - 1) attempt to autodetect if that enemy package is installed and active and then 2) add whatever rules are necessary to the enemy package's table (or the "filter" table if the enemy is still using iptables) in order to get our traffic through) I'm not sure where I'm going with this, other than to say that, on a bad day, trying to win this seems like a losing battle - let's say we add something to counteract docker's rules, and ufw's rules (and later when those packages add nftables support then we'll need to detect whether each is using iptables or nftables, and add counteracting rules for those as well); then what's next? Sorry, it's only Thursday and I'm already feeling defeated :-/ So what do you consider libvirt could do to make it acceptable to have nftables as the default backend on debian? Automatically add rules for the current state of what docker and ufw do? Or is there some other slightly more obscure package that we also need to compensate for before nftables backend is acceptable as default? (seriously, let's declare an enumerated list and then (hopefully, time permitting) take action on it. I would love to completely eliminate the iptables backend if I possibly could, and that certainly can't happen if some systems still have it as the default.
On Thu, Jan 30, 2025 at 01:20:40PM -0500, Laine Stump wrote:
On 1/30/25 10:48 AM, Andrea Bolognani wrote:
Debian 12 doesn't come with a new enough libvirt version anyway, but FYI a few months back I switched the default backend in Debian to nftables (matching Fedora) only to walk back the decision after getting several reports of it breaking software that's just too popular to ignore. See [1] for more details.
I don't expect that Debian will be able to move off the iptables backend any time soon, at least when it comes to the default. Changing the backend on a per-system basis is of course totally possible, as long as you understand the caveats.
Sigh.
In the days of iptables, there were 3 main tables (filter, nat, and mangle) and everybody's rules went into those same 3 tables. Within that single table, if a packet reached a REJECT or DROP rule before an ACCEPT rule (or the end of the table) then the packet would be dropped, but if it reached an ACCEPT rule first, then it would never see the REJECT rule, and be accepted.
But with nftables, there are an infinite number of "base tables", all traffic is evaluated against *all* tables *all the way* to either accept/reject in *all* cases, and it must get to the end of *every single table* without encountering a reject rule in order for the traffic to be accepted - there is no "early exit on accept" that skips all the rest of the tables if the traffic is accepted by one table.
[...]
There is no generic way to fix this problem. libvirt can't possible find every possible firewall system and add rules to the table of every single one that passes traffic from libvirt guests. I guess the best we can theoretically do is make a list of "supported firewall enemies" and add in extra stuff just like we currently do with firewalld - 1) attempt to autodetect if that enemy package is installed and active and then 2) add whatever rules are necessary to the enemy package's table (or the "filter" table if the enemy is still using iptables) in order to get our traffic through)
I'm not sure where I'm going with this, other than to say that, on a bad day, trying to win this seems like a losing battle - let's say we add something to counteract docker's rules, and ufw's rules (and later when those packages add nftables support then we'll need to detect whether each is using iptables or nftables, and add counteracting rules for those as well); then what's next? Sorry, it's only Thursday and I'm already feeling defeated :-/
If things really work the way you describe them, it sounds like an unsolvable problem indeed. Any scenario in which all possible components need to be aware of each other obviously doesn't scale. Have the nftables maintainers expressed their opinion about this? Surely they would have considered how to make filtering work without forcing extremely tight coupling.
So what do you consider libvirt could do to make it acceptable to have nftables as the default backend on debian? Automatically add rules for the current state of what docker and ufw do? Or is there some other slightly more obscure package that we also need to compensate for before nftables backend is acceptable as default? (seriously, let's declare an enumerated list and then (hopefully, time permitting) take action on it. I would love to completely eliminate the iptables backend if I possibly could, and that certainly can't happen if some systems still have it as the default.
I don't know if we can just draft a list. Docker and ufw would certainly need to work, but there might very well exist some other fairly popular package that I'm simply not aware of, for which being broken out of the box is not considered acceptable. That's exactly why tight coupling is undesirable. I'll note that the nwfilter driver not having an nftables backend is another, if secondary, reason to stick with iptables by default. The main goal for most people is to create a deployment that's completely free of the legacy userspace, and if some other driver is going to drag it in anyway, a big part of the benefit is immediately lost... -- Andrea Bolognani / Red Hat / Virtualization
On Thu, Jan 30, 2025 at 12:47:41PM -0800, Andrea Bolognani wrote:
On Thu, Jan 30, 2025 at 01:20:40PM -0500, Laine Stump wrote:
On 1/30/25 10:48 AM, Andrea Bolognani wrote:
Debian 12 doesn't come with a new enough libvirt version anyway, but FYI a few months back I switched the default backend in Debian to nftables (matching Fedora) only to walk back the decision after getting several reports of it breaking software that's just too popular to ignore. See [1] for more details.
I don't expect that Debian will be able to move off the iptables backend any time soon, at least when it comes to the default. Changing the backend on a per-system basis is of course totally possible, as long as you understand the caveats.
Sigh.
In the days of iptables, there were 3 main tables (filter, nat, and mangle) and everybody's rules went into those same 3 tables. Within that single table, if a packet reached a REJECT or DROP rule before an ACCEPT rule (or the end of the table) then the packet would be dropped, but if it reached an ACCEPT rule first, then it would never see the REJECT rule, and be accepted.
But with nftables, there are an infinite number of "base tables", all traffic is evaluated against *all* tables *all the way* to either accept/reject in *all* cases, and it must get to the end of *every single table* without encountering a reject rule in order for the traffic to be accepted - there is no "early exit on accept" that skips all the rest of the tables if the traffic is accepted by one table.
[...]
There is no generic way to fix this problem. libvirt can't possible find every possible firewall system and add rules to the table of every single one that passes traffic from libvirt guests. I guess the best we can theoretically do is make a list of "supported firewall enemies" and add in extra stuff just like we currently do with firewalld - 1) attempt to autodetect if that enemy package is installed and active and then 2) add whatever rules are necessary to the enemy package's table (or the "filter" table if the enemy is still using iptables) in order to get our traffic through)
I'm not sure where I'm going with this, other than to say that, on a bad day, trying to win this seems like a losing battle - let's say we add something to counteract docker's rules, and ufw's rules (and later when those packages add nftables support then we'll need to detect whether each is using iptables or nftables, and add counteracting rules for those as well); then what's next? Sorry, it's only Thursday and I'm already feeling defeated :-/
If things really work the way you describe them, it sounds like an unsolvable problem indeed. Any scenario in which all possible components need to be aware of each other obviously doesn't scale.
That's not quite the case. libvirt shouldn't need to know about docker, and vica-verca. docker & libvirt both need to know about the base OS' choice of firewall mgmt tool (ufw, firewalld, initscripts, etc) and support whichever the base OS has used. A decent number of variations, but not a combinatorial expansion at least.
Have the nftables maintainers expressed their opinion about this? Surely they would have considered how to make filtering work without forcing extremely tight coupling.
Usage is a decision for userspace and I believe the firewalld maintainers would expect everyone to directly use firewalld's APIs to achieve their goals and not go behind its back with native calls.
I'll note that the nwfilter driver not having an nftables backend is another, if secondary, reason to stick with iptables by default. The main goal for most people is to create a deployment that's completely free of the legacy userspace, and if some other driver is going to drag it in anyway, a big part of the benefit is immediately lost...
The nwfilter driver is not a big deal as its firewall rules are entirely self-contained and attached to the vnetXXX devices which no other tool will be trying to put rules on, so there's no expected clash & I've never heard any reported. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Feb 07, 2025 at 11:09:35AM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 30, 2025 at 12:47:41PM -0800, Andrea Bolognani wrote:
If things really work the way you describe them, it sounds like an unsolvable problem indeed. Any scenario in which all possible components need to be aware of each other obviously doesn't scale.
That's not quite the case. libvirt shouldn't need to know about docker, and vica-verca. docker & libvirt both need to know about the base OS' choice of firewall mgmt tool (ufw, firewalld, initscripts, etc) and support whichever the base OS has used. A decent number of variations, but not a combinatorial expansion at least.
If we can restrict the number of external components that we have to be mindful of to just firewall implementations, then things don't sound quite as bleak. Still quite a lot of work ahead. I'm wondering though, are we sure that e.g. Docker is doing the same thing? My understanding is that if we go through firewalld but they still add rules directly then we're screwed regardless.
Have the nftables maintainers expressed their opinion about this? Surely they would have considered how to make filtering work without forcing extremely tight coupling.
Usage is a decision for userspace and I believe the firewalld maintainers would expect everyone to directly use firewalld's APIs to achieve their goals and not go behind its back with native calls.
So the nftables design basically demands that an additional layer is added on top?
I'll note that the nwfilter driver not having an nftables backend is another, if secondary, reason to stick with iptables by default. The main goal for most people is to create a deployment that's completely free of the legacy userspace, and if some other driver is going to drag it in anyway, a big part of the benefit is immediately lost...
The nwfilter driver is not a big deal as its firewall rules are entirely self-contained and attached to the vnetXXX devices which no other tool will be trying to put rules on, so there's no expected clash & I've never heard any reported.
That's not what I meant. Some people are just very eager to not have iptables installed at all on their machines for whatever reason, and as long as one of the drivers can only use iptables as the backend that's much harder to achieve. -- Andrea Bolognani / Red Hat / Virtualization
On Fri, Feb 07, 2025 at 06:39:50AM -0800, Andrea Bolognani wrote:
On Fri, Feb 07, 2025 at 11:09:35AM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 30, 2025 at 12:47:41PM -0800, Andrea Bolognani wrote:
If things really work the way you describe them, it sounds like an unsolvable problem indeed. Any scenario in which all possible components need to be aware of each other obviously doesn't scale.
That's not quite the case. libvirt shouldn't need to know about docker, and vica-verca. docker & libvirt both need to know about the base OS' choice of firewall mgmt tool (ufw, firewalld, initscripts, etc) and support whichever the base OS has used. A decent number of variations, but not a combinatorial expansion at least.
If we can restrict the number of external components that we have to be mindful of to just firewall implementations, then things don't sound quite as bleak. Still quite a lot of work ahead.
I'm wondering though, are we sure that e.g. Docker is doing the same thing? My understanding is that if we go through firewalld but they still add rules directly then we're screwed regardless.
Yes, we can't solve it alone, if other apps still use direct rules, *and* their direct rules are applying broad DROP/REJECT rules. I don't know what docker adds, but if they're similar to libvirt their rules would be merely about opening holes, or restricting traffic on their own managed bridges, rather than blocking traffic broadly. In that case, docker would still be doomed by not using firewalld directly, but libvirt would be OK.
Have the nftables maintainers expressed their opinion about this? Surely they would have considered how to make filtering work without forcing extremely tight coupling.
Usage is a decision for userspace and I believe the firewalld maintainers would expect everyone to directly use firewalld's APIs to achieve their goals and not go behind its back with native calls.
So the nftables design basically demands that an additional layer is added on top?
IMHO that's unavoidable no matter what firewall technology is present in the kernel, because part of libvirt's need is to allow holes in a few places which are outside its own created bridge device.
I'll note that the nwfilter driver not having an nftables backend is another, if secondary, reason to stick with iptables by default. The main goal for most people is to create a deployment that's completely free of the legacy userspace, and if some other driver is going to drag it in anyway, a big part of the benefit is immediately lost...
The nwfilter driver is not a big deal as its firewall rules are entirely self-contained and attached to the vnetXXX devices which no other tool will be trying to put rules on, so there's no expected clash & I've never heard any reported.
That's not what I meant. Some people are just very eager to not have iptables installed at all on their machines for whatever reason, and as long as one of the drivers can only use iptables as the backend that's much harder to achieve.
RPM has choice deps eg "requires: nftables or iptables", but in the end we didn't use that in Fedora - we just left the default choice as the mandatory dep and didn't add a dep for the non-default. Can't quite remeber With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Feb 07, 2025 at 02:59:05PM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 07, 2025 at 06:39:50AM -0800, Andrea Bolognani wrote:
I'm wondering though, are we sure that e.g. Docker is doing the same thing? My understanding is that if we go through firewalld but they still add rules directly then we're screwed regardless.
Yes, we can't solve it alone, if other apps still use direct rules, *and* their direct rules are applying broad DROP/REJECT rules.
I don't know what docker adds, but if they're similar to libvirt their rules would be merely about opening holes, or restricting traffic on their own managed bridges, rather than blocking traffic broadly. In that case, docker would still be doomed by not using firewalld directly, but libvirt would be OK.
I'm not sure what Docker does either, but I can tell you for sure that, at least on Debian, switching libvirt to the nftables backend when Docker is installed makes guest connectivity break completely. Even if that turned out to be Docker's fault for not playing nice, the fact would remain that we can't default to a configuration that doesn't work when paired with such popular software. Things are different in Fedora because there's a clear indication that Podman and firewalld should be used instead of Docker and ufw. Debian is not nearly as opinionated as that.
Some people are just very eager to not have iptables installed at all on their machines for whatever reason, and as long as one of the drivers can only use iptables as the backend that's much harder to achieve.
RPM has choice deps eg "requires: nftables or iptables", but in the end we didn't use that in Fedora - we just left the default choice as the mandatory dep and didn't add a dep for the non-default. Can't quite remeber
Debian can do that too. The result is the same, because there is no choice when it comes to the nwfilter driver: iptables is the only available backend. So: # echo n \ | dnf --setopt=install_weak_deps=False install \ libvirt-daemon-driver-network \ 2>&1 | grep iptables iptables-libs x86_64 1.8.10-15.fc41 where iptables-libs is there only because dnsmasq requires it, but: # echo n \ | dnf --setopt=install_weak_deps=False install \ libvirt-daemon-driver-nwfilter \ 2>&1 | grep iptables iptables-legacy x86_64 1.8.10-15.fc41 iptables-legacy-libs x86_64 1.8.10-15.fc41 iptables-libs x86_64 1.8.10-15.fc41 And so obviously: # echo n \ | dnf --setopt=install_weak_deps=False install \ libvirt-daemon-kvm \ 2>&1 | grep iptables iptables-legacy x86_64 1.8.10-15.fc41 iptables-legacy-libs x86_64 1.8.10-15.fc41 iptables-libs x86_64 1.8.10-15.fc41 In other words, the only way to avoid iptables getting installed is to go out of your way and ensure that the nwfilter driver is left out. I'm not sure why iptables-legacy is considered instead of iptables-nft, but that's another topic. -- Andrea Bolognani / Red Hat / Virtualization
On Fri, Feb 07, 2025 at 07:44:02AM -0800, Andrea Bolognani wrote:
On Fri, Feb 07, 2025 at 02:59:05PM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 07, 2025 at 06:39:50AM -0800, Andrea Bolognani wrote:
I'm wondering though, are we sure that e.g. Docker is doing the same thing? My understanding is that if we go through firewalld but they still add rules directly then we're screwed regardless.
Yes, we can't solve it alone, if other apps still use direct rules, *and* their direct rules are applying broad DROP/REJECT rules.
I don't know what docker adds, but if they're similar to libvirt their rules would be merely about opening holes, or restricting traffic on their own managed bridges, rather than blocking traffic broadly. In that case, docker would still be doomed by not using firewalld directly, but libvirt would be OK.
I'm not sure what Docker does either, but I can tell you for sure that, at least on Debian, switching libvirt to the nftables backend when Docker is installed makes guest connectivity break completely.
Even if that turned out to be Docker's fault for not playing nice, the fact would remain that we can't default to a configuration that doesn't work when paired with such popular software.
Would be interesting to know what docker was doing to break it, as it might be something silly that's overlooked & easily fixed. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Feb 07, 2025 at 03:48:00PM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 07, 2025 at 07:44:02AM -0800, Andrea Bolognani wrote:
I'm not sure what Docker does either, but I can tell you for sure that, at least on Debian, switching libvirt to the nftables backend when Docker is installed makes guest connectivity break completely.
Even if that turned out to be Docker's fault for not playing nice, the fact would remain that we can't default to a configuration that doesn't work when paired with such popular software.
Would be interesting to know what docker was doing to break it, as it might be something silly that's overlooked & easily fixed.
I wouldn't even know where to start to figure that out, but for anyone interested reproducing the problem should be as easy as installing Debian testing, installing docker, and changing the libvirt network backend to nftables. -- Andrea Bolognani / Red Hat / Virtualization
On Fri, Feb 07, 2025 at 08:28:47AM -0800, Andrea Bolognani wrote:
On Fri, Feb 07, 2025 at 03:48:00PM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 07, 2025 at 07:44:02AM -0800, Andrea Bolognani wrote:
I'm not sure what Docker does either, but I can tell you for sure that, at least on Debian, switching libvirt to the nftables backend when Docker is installed makes guest connectivity break completely.
Even if that turned out to be Docker's fault for not playing nice, the fact would remain that we can't default to a configuration that doesn't work when paired with such popular software.
Would be interesting to know what docker was doing to break it, as it might be something silly that's overlooked & easily fixed.
I wouldn't even know where to start to figure that out, but for anyone interested reproducing the problem should be as easy as installing Debian testing, installing docker, and changing the libvirt network backend to nftables.
I normally debug by inserting "-j LOG" rules at random places until I find the rule that's blocking the traffic. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, 2025-02-07 at 16:44 +0000, Daniel P. Berrangé wrote:
On Fri, Feb 07, 2025 at 08:28:47AM -0800, Andrea Bolognani wrote:
On Fri, Feb 07, 2025 at 03:48:00PM +0000, Daniel P. Berrangé wrote:
On Fri, Feb 07, 2025 at 07:44:02AM -0800, Andrea Bolognani wrote:
I'm not sure what Docker does either, but I can tell you for sure that, at least on Debian, switching libvirt to the nftables backend when Docker is installed makes guest connectivity break completely.
Even if that turned out to be Docker's fault for not playing nice, the fact would remain that we can't default to a configuration that doesn't work when paired with such popular software.
Would be interesting to know what docker was doing to break it, as it might be something silly that's overlooked & easily fixed.
I wouldn't even know where to start to figure that out, but for anyone interested reproducing the problem should be as easy as installing Debian testing, installing docker, and changing the libvirt network backend to nftables.
I normally debug by inserting "-j LOG" rules at random places until I find the rule that's blocking the traffic.
I find the TRACE target very useful: TRACE This target marks packets so that the kernel will log every rule which match the packets as those traverse the tables, chains, rules. It can only be used in the raw table. kind regards Björn
On Thu, Jan 30, 2025 at 01:20:40PM -0500, Laine Stump wrote:
On 1/30/25 10:48 AM, Andrea Bolognani wrote:
On Wed, Jan 29, 2025 at 12:24:47PM -0500, Laine Stump wrote:
On 1/29/25 8:39 AM, oza.4h07@gmail.com wrote: (BTW, if your distro has libvirt 10.4.0 or newer, you can tell it to use nftables rules rather than iptables - just add:
firewall_backend = "nftables"
to /etc/libvirt/network.conf)
Debian 12 doesn't come with a new enough libvirt version anyway, but FYI a few months back I switched the default backend in Debian to nftables (matching Fedora) only to walk back the decision after getting several reports of it breaking software that's just too popular to ignore. See [1] for more details.
I don't expect that Debian will be able to move off the iptables backend any time soon, at least when it comes to the default. Changing the backend on a per-system basis is of course totally possible, as long as you understand the caveats.
Sigh.
In the days of iptables, there were 3 main tables (filter, nat, and mangle) and everybody's rules went into those same 3 tables. Within that single table, if a packet reached a REJECT or DROP rule before an ACCEPT rule (or the end of the table) then the packet would be dropped, but if it reached an ACCEPT rule first, then it would never see the REJECT rule, and be accepted.
But with nftables, there are an infinite number of "base tables", all traffic is evaluated against *all* tables *all the way* to either accept/reject in *all* cases, and it must get to the end of *every single table* without encountering a reject rule in order for the traffic to be accepted - there is no "early exit on accept" that skips all the rest of the tables if the traffic is accepted by one table.
(yeah, I know there's a lot of words enclosed in *..* there)
So the reason that traffic is flowing with libvirt's iptables backend + docker/whatever is that libvirt loads its rules *last* and so it can override the "other guy's" "REJECT iptables rulesby inserting its own ACCEPT rules at the beginning of the chain - the docket/whatever rules are never even encountered.
Also note that this only works if libvirt is very careful with its own rules to not REJECT traffic that other tools want to allow. We try to be good in this regard by tightly scoping our rules to subnets/NICs that we are responsible for.
But when libvirt uses the nftables backend, it creates its own base level table (just as firewalld does). If docker/whatever is still using iptables, its iptables rules are converted into nftables rules and added to a table named "filter". So now each packet is processed through the libvirt table up until it reaches a resolution, and then it's processed through the entire filter table until it reaches a resolution - if either of the tables leads to a reject result, then the traffic is dropped. The only way around this is to add mirrored ACCEPT rules to the "filter" table (ie where all the iptables-converted-to-nftables rules are located) that match the traffic libvirt wants to accept. If we do this, then we're just using iptables again, which is what we're trying to *get away from*.
And this isn't a temporary issue caused by docker/whatever remining on iptables - once they modernize and begin using nftables, it will be the same situation, except instead of the "other" table being "filter", it will be a different table created just for docker/whatever (e.g. called "docker") and we'll be back at the same problem.
(Note that this exact same problem will occur if, for example, someone installs docker on a system with firewalld or ufw or whatever. If you don't see problems in these cases, then its because one of the two packages is adding in extra rules to the other package's table to accept the traffic they want accepted)
There is no generic way to fix this problem.
The problem is inherant in using the low level firewall features directly. IMHO the "correct" way for everything to play nice is for both docker and libvirt to exclusively use higher level APIs from firewalld. firewalld will thus have everything in the same base tables and all will be good.... ...except this means we have to create backends for *every* distinct firewall tool that might exist. This sucks, but honestly that's the only general solution to the priorization problem.
libvirt can't possible find every possible firewall system and add rules to the table of every single one that passes traffic from libvirt guests. I guess the best we can theoretically do is make a list of "supported firewall enemies" and add in extra stuff just like we currently do with firewalld - 1) attempt to autodetect if that enemy package is installed and active and then 2) add whatever rules are necessary to the enemy package's table (or the "filter" table if the enemy is still using iptables) in order to get our traffic through)
Our extra injection of rules to "workaround" our "enemies" is at best a hack that will break at some point. Directly interacting with each firewall tool is the only sustainable option from a functional POV. Direct use of 'nftables' should be only a fallback for scenarios where we don't support a given FW tool, until such time as native support is added.
So what do you consider libvirt could do to make it acceptable to have nftables as the default backend on debian? Automatically add rules for the current state of what docker and ufw do? Or is there some other slightly more obscure package that we also need to compensate for before nftables backend is acceptable as default? (seriously, let's declare an enumerated list and then (hopefully, time permitting) take action on it. I would love to completely eliminate the iptables backend if I possibly could, and that certainly can't happen if some systems still have it as the default.
We need native support for 'ufw' and 'firewalld', and our "default" behaviour would be to auto-detect which backend to prefer. No single backend will ever work correctly, not even the legacy 'iptables' backend, we're just shuffling which subset of users is affected by brokenness. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thank you all very much,for replying. Can I ask the following questions : 1. are the fw rules added by libvirtd, hardcoded in libvirtd source code or editable ? 2. can a sys admin enable ou disable these rules inclusion (ie set a paramater somewhere so that these rules are or are not included in currently running firewall) ? Given the variety of roles a Linux host can run, having a single set of rules to fit all these roles is impossible. Giving sys admins the keys (documentation, examples, ...) to let them tune whatever they need is enough.
On Fri, Jan 31, 2025 at 09:26:32AM -0000, oza.4h07@gmail.com wrote:
Thank you all very much,for replying.
Can I ask the following questions :
1. are the fw rules added by libvirtd, hardcoded in libvirtd source code or editable ?
They are defined by our source code.
2. can a sys admin enable ou disable these rules inclusion (ie set a paramater somewhere so that these rules are or are not included in currently running firewall) ?
The default <forward> config will always add rules, since without adding rules you don't get any functional connectivity for guests. If you want to take full responsibility for adding rules you can change the cnofig to <forward mode='open'/> which will give you a broken connectivity until you add your own rules. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (5)
-
Andrea Bolognani -
Björn Lässig -
Daniel P. Berrangé -
Laine Stump -
oza.4h07＠gmail.com