On 2/25/25 1:58 AM, Robin Lee Powell wrote:
On Mon, Feb 24, 2025 at 04:25:58PM -0500, Laine Stump wrote:
> On 2/21/25 7:02 PM, robinleepowell(a)gmail.com wrote:
>> So I, like many other people, have hit problems with nftables ordering, as has
been discussed on this mailing list MANY TIMES.
>>
>> This whole thing seemed ridiculous so I asked the nftables people about what one
is *supposed* to do in this situation. It turns out that the standard solution is for
libvirt's nftables rules to set a packet mark (there's a collision possibility
here but it's a 32 bit integer if you pick one at random it shouldn't be a
problem) and then the user adds a rule to exclude packets with that mark from any reject
rules they might have, or explicitly accept marked packets in their own chains, or
whatever.
>
> Was the discussion on a public forum somewhere? I'd like to look at exactly
> what they said.
Yep! Sorry, thought I linked to it, oops.
https://lore.kernel.org/netfilter/132daf73-668f-4321-8945-c809db2277f5@re...
Thanks! (I'm surprised I've never subscribed to that list, so that it
could be yet another folder with ever-increasing number of unread
messages :-/. Seriously though, I probably should be paying attention to it)
My one comment about their response/advice is that, while they suggest
that libvirt shouldn't be adding any firewall rules but should instead
just have docs telling people what rules they need to add, the entire
purpose of libvirt's virtual networks was to provide essentially a
single "That was Easy" button that can be pushed which sets up
*everything* needed for a guest to communicate with the outside - the
user can just say "create a network using NAT forwarding" and libvirt
creates the bridge device, sets up the DHCP and DNS servers listening on
that bridge device, turns on IP forwarding (if it's off) and also adds
all the rules necessary to allow the traffic to pass (and to NAT the
packets if that was requested). If we did "all those things *except* the
firewall rules" it would add another hurdle for inexperienced users to
get their VMs fully functional. (BTW, if someone wants that, they can
just define the network with <forward mode='open'/> and it will setup
everything except the firewall rules).
In my opinion, libvirt definitely *should* be adding the appropriate
rules (for the types of networks that are meant to be the most "plug and
play"), but also (and this is where the netfilter people are correct)
should do a better job of documenting what might need to be done beyond
that on some systems. In the meantime we should do our best to automate
(or at least specifically document) the "what might need to be done" for
as many other utilities as possible (for example the way we put all our
bridge devices in a special firewalld zone that we automatically add to
firewalld's configuration).
>> It's not *as nice* as the iptables situation, but having
documentation that says "if you're using nftables make sure that packets with
mark 79892 are accepted in all your chains" is quite straightforward compared to the
current situation of "LOL good luck". (I'm not blaming anyone there!, the
current situation is impossible for libvirt to navigate and it's not anyone's
fault.)
>
> It does still require that the other utilities know this secret number, and
> agree to "anti-reject" it as we've requested, though. Also doesn't
this
> require that libvirt's table is processed first, before the other utilities'
> tables? Otherwise, if the other tables are traversed before libvirt has a
> chance to mark the packet with the special number, they won't get the
> signal, so they'll reject the traffic. So I we would have set our table as a
> higher priority, but then what if someone else sets their table with an
> *even higher* priority? e.g. firewalld has "priority filter + 10" for its
> forwarding rules, so could make ours "priority filter + 20", but what if,
> e.g. docker decided to make theirs "priority filter +50"?). (yes,
that's all
> a rhetorical question. I guess in the end everything like this that we do
> will chip away a bit more at the list of people who encounter problems; it
> will never reach 0, but it will at least get closer :-))
Yep, those are all real concerns. :sigh:
> Aside from that, libvirt's nftables rules are default accept, and it has no
> rules looking at traffic that is destined for the host, only for forwarded
> traffic that is going *through* the host, mainly with the intent of
> rejecting stuff it doesn't like. So are you/they suggesting that this
> forwarded traffic be marked with the special "libvirt code"? Or that we
> should also add back rules that match input DNS/DHCP/TFTP on the
> libvirt-created bridges, and have them both accept and mark those packets?
I think it'd have to be the latter to actually work.
>> If y'all don't like that, what's working excellently for me is adding
`iifname "virbr*" accept` to my rule chain. FWIW.
>
> Just keep in mind that "iifname" has to fetch the name of the interface
and
> do a string comparison for each packet, while "iif" just does a quick
> comparison of ifindex, which I think is already saved away in the skb (of
> course wildcards aren't possible in that case, but if you have just a couple
> of libvirt networks it's still more efficient to have a rule using
"iif" for
> each interface.
The reason I have to use iifname is that at the time my rules are
loaded, the virbr interfaces *don't exist*. Like I actually have no
choice; it won't work the other way, unless I'm badly missing
something.
Yeah, good point. One alternate method you could experiment with if you
thought that the string compare was having anything more than a .0005%
effect on performance would be to use a libvirt network hook script
(
https://libvirt.org/hooks.html ) - if you put your rule additions in
/etc/libvirt/hooks/network in the section run when argv[2] is "started"
then the bridge device (which you could grab out of the XML provided to
the script on stdin) would already be created. The down side is that
once you've added a hook script, all your networks are marked as
"tainted" which could create supportability problems if you were paying
someone for support; since you're using Fedora that's not a problem
though :-)
(I do recall someone at one point posting a perf chart where it showed a
significant amount of the kernel time on their system was spent doing
strcmp of the interface names. I don't remember exactly how much or
where it was posted, but I do remember being surprised at how much CPU
was used up for that...)
>> It was very hard to navigate through this situation because there's no
documentation that this problem even exists.
>
> Yeah, that's my fault. When I added the nftables backend, I forgot to update
>
https://libvirt.org/firewall.html (which is in docs/firewall.rst in the
> libvirt sources). (also at the time I wrote the code, I I keep remembering
> that I should do that, but only when I'm in the middle of something else and
> somehow I haven't managed to even write it down on a list.
(I'm pretty sure there is a missing partial sentence in between the "I
I" up there, but no idea what that partial sentence was, which is an
example of the attention span that has led to (for example) this lack of
proper documentation :-P)
No attack intended; FOSS work is hard. :)
None perceived :-). Over the years there have been (fortunately not too
many, but certainly more than enough!) irate and hostile messages that
have gone by on this and other lists, but yours have not been in that
class - to the contrary, you're providing the necessary information for
us to understand your problem, doing your own investigation to help
narrow it down and better define it, and even volunteering to do some of
the work to fix it - that's the definition of being a good FOSS
community citizen :-) (oh, and I forgot the part about not
indiscriminately hurling insults around :-P)
>> My suggestion is to describe the situation at
https://libvirt.org/firewall.html
and suggest the virbr* fix, and down the road maybe look at this mark thing.
>
> That's a kind of a broad solution though - libvirt's rules only reject
> specific traffic between libvirt-created bridges (and incoming traffic from
> outside a bridge's direct connects in the case of forward mode='nat'),
> Anywhere they allow traffic, they allow *all* of it. The real problematic
> stuff is traffic between the guests and the host (the rules we've had for
> iptables that are absent in nftables are those to allow inbound DNS, DHCP,
> and TFTP that are arriving on a virbr* interface, and destined for the
> host). If you allow *all* traffic for virbr*, then you're leaving the host
> wide open to all traffic from any guests (since libvirt's own rules are
> default accept). I think the suggestion needs to be more than just "allow
> all incoming on virbr*".
That's fair; I suppose we could post something equivalent to the old
iptables rules?
Yeah, and say something like "for minimal functionality and connection
to the outside"
And I do also like the idea of marking all the traffic that libvirt's
rules accept (and adding back the rules / marking the packets for the
basic functionality that our network setup provides). While it may not
work for everyone, it would likely be easier for someone to use the info
"accept all traffic with mark xyzzy" than to give out a list of all the
things that need to be accepted (and also more efficient, since there
would then be only a single extra rule in the other table no matter how
many libvirt networks were active).
I'll add this to my list of things to do and try to get to it this week
while it's fresh in my mind; I'll be sure to Cc you on the patches so
you'll know when there is something available to experiment with.
>> I'd like to help. I'm happy to write up issues for this, and I'm
happy to write the updates to the firewall docs; just tell me what you'd like me to
do.
>
> firewall.rst should really be a shortened intro that links to the current
> firewall.html for iptables (maybe renaming it "iptables.rst/html"?), and
to
> a new nftables.rst/html for information about nftables (including an
> explanation of the "many tables, all must resolve to 'accept' problem.)
>
> Since I've never gotten around to it in spite of wanting it done, I'd
> certainly be happy to review an update done by anyone else :-)
Acknowledged. :)