On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote:

On 2/21/25 7:02 PM, robinleepowell@gmail.com wrote:

...

So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES.

This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever.

Was the discussion on a public forum somewhere? I'd like to look at exactly what they said.

Yep! Sorry, thought I linked to it, oops. https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@redha...

...

It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.)

It does still require that the other utilities know this secret number, and agree to "anti-reject" it as we've requested, though. Also doesn't this require that libvirt's table is processed first, before the other utilities' tables? Otherwise, if the other tables are traversed before libvirt has a chance to mark the packet with the special number, they won't get the signal, so they'll reject the traffic. So I we would have set our table as a higher priority, but then what if someone else sets their table with an *even higher* priority? e.g. firewalld has "priority filter + 10" for its forwarding rules, so could make ours "priority filter + 20", but what if, e.g. docker decided to make theirs "priority filter +50"?). (yes, that's all a rhetorical question. I guess in the end everything like this that we do will chip away a bit more at the list of people who encounter problems; it will never reach 0, but it will at least get closer :-))

Yep, those are all real concerns. :sigh:

Aside from that, libvirt's nftables rules are default accept, and it has no rules looking at traffic that is destined for the host, only for forwarded traffic that is going *through* the host, mainly with the intent of rejecting stuff it doesn't like. So are you/they suggesting that this forwarded traffic be marked with the special "libvirt code"? Or that we should also add back rules that match input DNS/DHCP/TFTP on the libvirt-created bridges, and have them both accept and mark those packets?

I think it'd have to be the latter to actually work.

...

If y'all don't like that, what's working excellently for me is adding `iifname "virbr*" accept` to my rule chain. FWIW.

Just keep in mind that "iifname" has to fetch the name of the interface and do a string comparison for each packet, while "iif" just does a quick comparison of ifindex, which I think is already saved away in the skb (of course wildcards aren't possible in that case, but if you have just a couple of libvirt networks it's still more efficient to have a rule using "iif" for each interface.

The reason I have to use iifname is that at the time my rules are loaded, the virbr interfaces *don't exist*. Like I actually have no choice; it won't work the other way, unless I'm badly missing something.

...

It was very hard to navigate through this situation because there's no documentation that this problem even exists.

Yeah, that's my fault. When I added the nftables backend, I forgot to update https://libvirt.org/firewall.html (which is in docs/firewall.rst in the libvirt sources). (also at the time I wrote the code, I I keep remembering that I should do that, but only when I'm in the middle of something else and somehow I haven't managed to even write it down on a list.

No attack intended; FOSS work is hard. :)

...

My suggestion is to describe the situation at https://libvirt.org/firewall.html and suggest the virbr* fix, and down the road maybe look at this mark thing.

That's a kind of a broad solution though - libvirt's rules only reject specific traffic between libvirt-created bridges (and incoming traffic from outside a bridge's direct connects in the case of forward mode='nat'), Anywhere they allow traffic, they allow *all* of it. The real problematic stuff is traffic between the guests and the host (the rules we've had for iptables that are absent in nftables are those to allow inbound DNS, DHCP, and TFTP that are arriving on a virbr* interface, and destined for the host). If you allow *all* traffic for virbr*, then you're leaving the host wide open to all traffic from any guests (since libvirt's own rules are default accept). I think the suggestion needs to be more than just "allow all incoming on virbr*".

That's fair; I suppose we could post something equivalent to the old iptables rules?

...

I'd like to help. I'm happy to write up issues for this, and I'm happy to write the updates to the firewall docs; just tell me what you'd like me to do.

firewall.rst should really be a shortened intro that links to the current firewall.html for iptables (maybe renaming it "iptables.rst/html"?), and to a new nftables.rst/html for information about nftables (including an explanation of the "many tables, all must resolve to 'accept' problem.)

Since I've never gotten around to it in spite of wanting it done, I'd certainly be happy to review an update done by anyone else :-)

Acknowledged. :)

On 2/25/25 1:58 AM, Robin Lee Powell wrote:

On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote:

...

On 2/21/25 7:02 PM, robinleepowell@gmail.com wrote:

...

So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES.

This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever.

Was the discussion on a public forum somewhere? I'd like to look at exactly what they said.

Yep! Sorry, thought I linked to it, oops. https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@redha...

Thanks! (I'm surprised I've never subscribed to that list, so that it could be yet another folder with ever-increasing number of unread messages :-/. Seriously though, I probably should be paying attention to it) My one comment about their response/advice is that, while they suggest that libvirt shouldn't be adding any firewall rules but should instead just have docs telling people what rules they need to add, the entire purpose of libvirt's virtual networks was to provide essentially a single "That was Easy" button that can be pushed which sets up *everything* needed for a guest to communicate with the outside - the user can just say "create a network using NAT forwarding" and libvirt creates the bridge device, sets up the DHCP and DNS servers listening on that bridge device, turns on IP forwarding (if it's off) and also adds all the rules necessary to allow the traffic to pass (and to NAT the packets if that was requested). If we did "all those things *except* the firewall rules" it would add another hurdle for inexperienced users to get their VMs fully functional. (BTW, if someone wants that, they can just define the network with <forward mode='open'/> and it will setup everything except the firewall rules). In my opinion, libvirt definitely *should* be adding the appropriate rules (for the types of networks that are meant to be the most "plug and play"), but also (and this is where the netfilter people are correct) should do a better job of documenting what might need to be done beyond that on some systems. In the meantime we should do our best to automate (or at least specifically document) the "what might need to be done" for as many other utilities as possible (for example the way we put all our bridge devices in a special firewalld zone that we automatically add to firewalld's configuration).

...

...

It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.)

It does still require that the other utilities know this secret number, and agree to "anti-reject" it as we've requested, though. Also doesn't this require that libvirt's table is processed first, before the other utilities' tables? Otherwise, if the other tables are traversed before libvirt has a chance to mark the packet with the special number, they won't get the signal, so they'll reject the traffic. So I we would have set our table as a higher priority, but then what if someone else sets their table with an *even higher* priority? e.g. firewalld has "priority filter + 10" for its forwarding rules, so could make ours "priority filter + 20", but what if, e.g. docker decided to make theirs "priority filter +50"?). (yes, that's all a rhetorical question. I guess in the end everything like this that we do will chip away a bit more at the list of people who encounter problems; it will never reach 0, but it will at least get closer :-))

Yep, those are all real concerns. :sigh:

...

Aside from that, libvirt's nftables rules are default accept, and it has no rules looking at traffic that is destined for the host, only for forwarded traffic that is going *through* the host, mainly with the intent of rejecting stuff it doesn't like. So are you/they suggesting that this forwarded traffic be marked with the special "libvirt code"? Or that we should also add back rules that match input DNS/DHCP/TFTP on the libvirt-created bridges, and have them both accept and mark those packets?

I think it'd have to be the latter to actually work.

...

...

If y'all don't like that, what's working excellently for me is adding `iifname "virbr*" accept` to my rule chain. FWIW.

Just keep in mind that "iifname" has to fetch the name of the interface and do a string comparison for each packet, while "iif" just does a quick comparison of ifindex, which I think is already saved away in the skb (of course wildcards aren't possible in that case, but if you have just a couple of libvirt networks it's still more efficient to have a rule using "iif" for each interface.

The reason I have to use iifname is that at the time my rules are loaded, the virbr interfaces *don't exist*. Like I actually have no choice; it won't work the other way, unless I'm badly missing something.

Yeah, good point. One alternate method you could experiment with if you thought that the string compare was having anything more than a .0005% effect on performance would be to use a libvirt network hook script (https://libvirt.org/hooks.html ) - if you put your rule additions in /etc/libvirt/hooks/network in the section run when argv[2] is "started" then the bridge device (which you could grab out of the XML provided to the script on stdin) would already be created. The down side is that once you've added a hook script, all your networks are marked as "tainted" which could create supportability problems if you were paying someone for support; since you're using Fedora that's not a problem though :-) (I do recall someone at one point posting a perf chart where it showed a significant amount of the kernel time on their system was spent doing strcmp of the interface names. I don't remember exactly how much or where it was posted, but I do remember being surprised at how much CPU was used up for that...)

...

...

It was very hard to navigate through this situation because there's no documentation that this problem even exists.

Yeah, that's my fault. When I added the nftables backend, I forgot to update https://libvirt.org/firewall.html (which is in docs/firewall.rst in the libvirt sources). (also at the time I wrote the code, I I keep remembering that I should do that, but only when I'm in the middle of something else and somehow I haven't managed to even write it down on a list.

(I'm pretty sure there is a missing partial sentence in between the "I I" up there, but no idea what that partial sentence was, which is an example of the attention span that has led to (for example) this lack of proper documentation :-P)

No attack intended; FOSS work is hard. :)

None perceived :-). Over the years there have been (fortunately not too many, but certainly more than enough!) irate and hostile messages that have gone by on this and other lists, but yours have not been in that class - to the contrary, you're providing the necessary information for us to understand your problem, doing your own investigation to help narrow it down and better define it, and even volunteering to do some of the work to fix it - that's the definition of being a good FOSS community citizen :-) (oh, and I forgot the part about not indiscriminately hurling insults around :-P)

...

...

My suggestion is to describe the situation at https://libvirt.org/firewall.html and suggest the virbr* fix, and down the road maybe look at this mark thing.

That's a kind of a broad solution though - libvirt's rules only reject specific traffic between libvirt-created bridges (and incoming traffic from outside a bridge's direct connects in the case of forward mode='nat'), Anywhere they allow traffic, they allow *all* of it. The real problematic stuff is traffic between the guests and the host (the rules we've had for iptables that are absent in nftables are those to allow inbound DNS, DHCP, and TFTP that are arriving on a virbr* interface, and destined for the host). If you allow *all* traffic for virbr*, then you're leaving the host wide open to all traffic from any guests (since libvirt's own rules are default accept). I think the suggestion needs to be more than just "allow all incoming on virbr*".

That's fair; I suppose we could post something equivalent to the old iptables rules?

Yeah, and say something like "for minimal functionality and connection to the outside" And I do also like the idea of marking all the traffic that libvirt's rules accept (and adding back the rules / marking the packets for the basic functionality that our network setup provides). While it may not work for everyone, it would likely be easier for someone to use the info "accept all traffic with mark xyzzy" than to give out a list of all the things that need to be accepted (and also more efficient, since there would then be only a single extra rule in the other table no matter how many libvirt networks were active). I'll add this to my list of things to do and try to get to it this week while it's fresh in my mind; I'll be sure to Cc you on the patches so you'll know when there is something available to experiment with.

...

...

I'd like to help. I'm happy to write up issues for this, and I'm happy to write the updates to the firewall docs; just tell me what you'd like me to do.

firewall.rst should really be a shortened intro that links to the current firewall.html for iptables (maybe renaming it "iptables.rst/html"?), and to a new nftables.rst/html for information about nftables (including an explanation of the "many tables, all must resolve to 'accept' problem.)

Since I've never gotten around to it in spite of wanting it done, I'd certainly be happy to review an update done by anyone else :-)

Acknowledged. :)

On Tue, Feb 25, 2025 at 02:26:44PM +0000, Daniel P. Berrangé wrote:

On Tue, Feb 25, 2025 at 09:20:41AM -0500, Laine Stump wrote:

...

On 2/25/25 1:58 AM, Robin Lee Powell wrote:

...

On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote:

...

On 2/21/25 7:02 PM, robinleepowell@gmail.com wrote:

...

So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES.

This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever.

Was the discussion on a public forum somewhere? I'd like to look at exactly what they said.

Yep! Sorry, thought I linked to it, oops. https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@redha...

Thanks! (I'm surprised I've never subscribed to that list, so that it could be yet another folder with ever-increasing number of unread messages :-/. Seriously though, I probably should be paying attention to it)

My one comment about their response/advice is that, while they suggest that libvirt shouldn't be adding any firewall rules but should instead just have docs telling people what rules they need to add, the entire purpose of libvirt's virtual networks was to provide essentially a single "That was Easy" button that can be pushed which sets up *everything* needed for a guest to communicate with the outside - the user can just say "create a network using NAT forwarding" and libvirt creates the bridge device, sets up the DHCP and DNS servers listening on that bridge device, turns on IP forwarding (if it's off) and also adds all the rules necessary to allow the traffic to pass (and to NAT the packets if that was requested). If we did "all those things *except* the firewall rules" it would add another hurdle for inexperienced users to get their VMs fully functional. (BTW, if someone wants that, they can just define the network with <forward mode='open'/> and it will setup everything except the firewall rules).

Yes, that comment is really in denial about what users expect. Giving them a "bag of bits" and expecting them to assemble a working solution manually from the docs is not credible.

We need this networking to "just work" out of the box. The virtual networking is/was one of the single biggest things that makes libvirt easy to use when needing serious networking capabilities, compared to direct use of QEMU (at least prior to passt, since slirp was never for serious use)

I had exactly the same response as both of you. :) Which is why I didn't bother replying to that part.