On 1/29/25 8:39 AM, oza.4h07(a)gmail.com wrote: I assume you are actually using libvirt's "default" virtual network, and that the rules you want to add are related to that network? If the rules are unrelated, you can set them up however / wherever you like. But it these rule are directly related to the virtual network created by libvirt, then take a look at https://libvirt.org/hooks.html You can add extra rules by creating the file /etc/libvirt/hooks/network (which is a shell script) and running the appropriate nftables/iptables commands there. The above linked page will show you how to construct the script so that commands to add rules (or other stuff) are run when the network starts, and other commands to remove/undo the rules/stuff are run when the network is stopped. (BTW, if your distro has libvirt 10.4.0 or newer, you can tell it to use nftables rules rather than iptables - just add: firewall_backend = "nftables" to /etc/libvirt/network.conf)

On 1/30/25 10:48 AM, Andrea Bolognani wrote: Sigh. In the days of iptables, there were 3 main tables (filter, nat, and mangle) and everybody's rules went into those same 3 tables. Within that single table, if a packet reached a REJECT or DROP rule before an ACCEPT rule (or the end of the table) then the packet would be dropped, but if it reached an ACCEPT rule first, then it would never see the REJECT rule, and be accepted. But with nftables, there are an infinite number of "base tables", all traffic is evaluated against *all* tables *all the way* to either accept/reject in *all* cases, and it must get to the end of *every single table* without encountering a reject rule in order for the traffic to be accepted - there is no "early exit on accept" that skips all the rest of the tables if the traffic is accepted by one table. (yeah, I know there's a lot of words enclosed in *..* there) So the reason that traffic is flowing with libvirt's iptables backend + docker/whatever is that libvirt loads its rules *last* and so it can override the "other guy's" "REJECT iptables rulesby inserting its own ACCEPT rules at the beginning of the chain - the docket/whatever rules are never even encountered. But when libvirt uses the nftables backend, it creates its own base level table (just as firewalld does). If docker/whatever is still using iptables, its iptables rules are converted into nftables rules and added to a table named "filter". So now each packet is processed through the libvirt table up until it reaches a resolution, and then it's processed through the entire filter table until it reaches a resolution - if either of the tables leads to a reject result, then the traffic is dropped. The only way around this is to add mirrored ACCEPT rules to the "filter" table (ie where all the iptables-converted-to-nftables rules are located) that match the traffic libvirt wants to accept. If we do this, then we're just using iptables again, which is what we're trying to *get away from*. And this isn't a temporary issue caused by docker/whatever remining on iptables - once they modernize and begin using nftables, it will be the same situation, except instead of the "other" table being "filter", it will be a different table created just for docker/whatever (e.g. called "docker") and we'll be back at the same problem. (Note that this exact same problem will occur if, for example, someone installs docker on a system with firewalld or ufw or whatever. If you don't see problems in these cases, then its because one of the two packages is adding in extra rules to the other package's table to accept the traffic they want accepted) There is no generic way to fix this problem. libvirt can't possible find every possible firewall system and add rules to the table of every single one that passes traffic from libvirt guests. I guess the best we can theoretically do is make a list of "supported firewall enemies" and add in extra stuff just like we currently do with firewalld - 1) attempt to autodetect if that enemy package is installed and active and then 2) add whatever rules are necessary to the enemy package's table (or the "filter" table if the enemy is still using iptables) in order to get our traffic through) I'm not sure where I'm going with this, other than to say that, on a bad day, trying to win this seems like a losing battle - let's say we add something to counteract docker's rules, and ufw's rules (and later when those packages add nftables support then we'll need to detect whether each is using iptables or nftables, and add counteracting rules for those as well); then what's next? Sorry, it's only Thursday and I'm already feeling defeated :-/ So what do you consider libvirt could do to make it acceptable to have nftables as the default backend on debian? Automatically add rules for the current state of what docker and ufw do? Or is there some other slightly more obscure package that we also need to compensate for before nftables backend is acceptable as default? (seriously, let's declare an enumerated list and then (hopefully, time permitting) take action on it. I would love to completely eliminate the iptables backend if I possibly could, and that certainly can't happen if some systems still have it as the default.

On Thu, Jan 30, 2025 at 12:47:41PM -0800, Andrea Bolognani wrote: That's not quite the case. libvirt shouldn't need to know about docker, and vica-verca. docker & libvirt both need to know about the base OS' choice of firewall mgmt tool (ufw, firewalld, initscripts, etc) and support whichever the base OS has used. A decent number of variations, but not a combinatorial expansion at least. Usage is a decision for userspace and I believe the firewalld maintainers would expect everyone to directly use firewalld's APIs to achieve their goals and not go behind its back with native calls. The nwfilter driver is not a big deal as its firewall rules are entirely self-contained and attached to the vnetXXX devices which no other tool will be trying to put rules on, so there's no expected clash & I've never heard any reported. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Feb 07, 2025 at 06:39:50AM -0800, Andrea Bolognani wrote: Yes, we can't solve it alone, if other apps still use direct rules, *and* their direct rules are applying broad DROP/REJECT rules. I don't know what docker adds, but if they're similar to libvirt their rules would be merely about opening holes, or restricting traffic on their own managed bridges, rather than blocking traffic broadly. In that case, docker would still be doomed by not using firewalld directly, but libvirt would be OK. IMHO that's unavoidable no matter what firewall technology is present in the kernel, because part of libvirt's need is to allow holes in a few places which are outside its own created bridge device. RPM has choice deps eg "requires: nftables or iptables", but in the end we didn't use that in Fedora - we just left the default choice as the mandatory dep and didn't add a dep for the non-default. Can't quite remeber With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Feb 07, 2025 at 07:44:02AM -0800, Andrea Bolognani wrote: Would be interesting to know what docker was doing to break it, as it might be something silly that's overlooked & easily fixed. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Feb 07, 2025 at 08:28:47AM -0800, Andrea Bolognani wrote: I normally debug by inserting "-j LOG" rules at random places until I find the rule that's blocking the traffic. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Jan 30, 2025 at 01:20:40PM -0500, Laine Stump wrote: Also note that this only works if libvirt is very careful with its own rules to not REJECT traffic that other tools want to allow. We try to be good in this regard by tightly scoping our rules to subnets/NICs that we are responsible for. The problem is inherant in using the low level firewall features directly. IMHO the "correct" way for everything to play nice is for both docker and libvirt to exclusively use higher level APIs from firewalld. firewalld will thus have everything in the same base tables and all will be good.... ...except this means we have to create backends for *every* distinct firewall tool that might exist. This sucks, but honestly that's the only general solution to the priorization problem. Our extra injection of rules to "workaround" our "enemies" is at best a hack that will break at some point. Directly interacting with each firewall tool is the only sustainable option from a functional POV. Direct use of 'nftables' should be only a fallback for scenarios where we don't support a given FW tool, until such time as native support is added. We need native support for 'ufw' and 'firewalld', and our "default" behaviour would be to auto-detect which backend to prefer. No single backend will ever work correctly, not even the legacy 'iptables' backend, we're just shuffling which subset of users is affected by brokenness. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Jan 31, 2025 at 09:26:32AM -0000, oza.4h07(a)gmail.com wrote: They are defined by our source code. The default <forward> config will always add rules, since without adding rules you don't get any functional connectivity for guests. If you want to take full responsibility for adding rules you can change the cnofig to <forward mode='open'/> which will give you a broken connectivity until you add your own rules. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|