nftables rules (DNS, DHCP, etc) not being written on Fedora 41
I do not think this is any of the similar issues that have been posted to this list; I've checked. In particular, this in *NOT* an issue with nft rules precedence; the rules are simply not being written by libvirt. I'm running vagrant which is running libvirt on a Fedora 41 host. I do not think this is a vagrant problem, but if people want me to run virsh commands directly I certainly can. Anyway, vagrant was failing until I noticed that the default for /etc/libvirt/network.conf is now nftables, which I did not have set up. When I set `firewall_backend = "iptables"`, everything worked fine. I want to emphasize that: the same vagrant / libvirt setup *was* working with iptables. But I took that as a sign that it was time to move to nftables, so I moved everything on this host and stuff is back to working. But libvirt just isn't writing out the right nft rules. Like, at all. Here's the network vagrant creates: $ sudo virsh net-dumpxml vagrant-libvirt <network connections='1' ipv6='yes'> <name>vagrant-libvirt</name> <uuid>b2d93ef4-b305-4382-a380-c1eca92d8ebd</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:63:d3:f6'/> <ip address='192.168.121.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.121.1' end='192.168.121.254'/> </dhcp> </ip> </network> And here's the *entire* libvirt-related ruleset in nftables: table ip libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output } chain guest_output { ip saddr 192.168.121.0/24 iif "virbr1" counter packets 0 bytes 0 accept iif "virbr1" counter packets 0 bytes 0 reject } chain guest_input { oif "virbr1" ip daddr 192.168.121.0/24 ct state established,related counter packets 0 bytes 0 accept oif "virbr1" counter packets 0 bytes 0 reject } chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept } chain guest_nat { type nat hook postrouting priority srcnat; policy accept; ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter packets 1 bytes 187 return ip saddr 192.168.121.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 meta l4proto udp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade } } table ip6 libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output } chain guest_output { iif "virbr1" counter packets 0 bytes 0 reject } chain guest_input { oif "virbr1" counter packets 0 bytes 0 reject } chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept } chain guest_nat { type nat hook postrouting priority srcnat; policy accept; } } Here's some log output (I have virtnetworkd running with -v): $ sudo journalctl -u virtnetworkd.service | grep -i nft [snip] Feb 20 21:36:42 stodi.digitalkingdom.org virtnetworkd[89119]: using firewall_backend: 'nftables' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip6 libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output ip saddr 192.168.121.0/24 iif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 ip daddr 192.168.121.0/24 ct state related,established counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto udp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 255.255.255.255/32 counter return' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter return' So it's not like trying to create the rules and failing, that I can see; it just isn't trying. To clarify this setup appears to be missing every rule that would be needed for DHCPv4 and DNS to work. Like https://gitlab.com/libvirt/libvirt/-/issues/88#note_694493261 shows rules for udp 67 and 53 that are just 100% not being created at all. I have no idea what's going wrong here or even where to look; the code at https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/GT75X... sure *looks like* it should be unconditionally adding those rules. Help?
Setting `firewall_backend = "iptables"` and allowing it to use the iptables-nft shim *did* get the right rules into place. I then ran smack into the "my main input chain has a default deny rule", but that's my problem. :) On Thu, Feb 20, 2025 at 10:33 PM <robinleepowell@gmail.com> wrote:
I do not think this is any of the similar issues that have been posted to this list; I've checked.
In particular, this in *NOT* an issue with nft rules precedence; the rules are simply not being written by libvirt.
I'm running vagrant which is running libvirt on a Fedora 41 host. I do not think this is a vagrant problem, but if people want me to run virsh commands directly I certainly can.
Anyway, vagrant was failing until I noticed that the default for /etc/libvirt/network.conf is now nftables, which I did not have set up. When I set `firewall_backend = "iptables"`, everything worked fine.
I want to emphasize that: the same vagrant / libvirt setup *was* working with iptables.
But I took that as a sign that it was time to move to nftables, so I moved everything on this host and stuff is back to working.
But libvirt just isn't writing out the right nft rules. Like, at all.
Here's the network vagrant creates:
$ sudo virsh net-dumpxml vagrant-libvirt <network connections='1' ipv6='yes'> <name>vagrant-libvirt</name> <uuid>b2d93ef4-b305-4382-a380-c1eca92d8ebd</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:63:d3:f6'/> <ip address='192.168.121.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.121.1' end='192.168.121.254'/> </dhcp> </ip> </network>
And here's the *entire* libvirt-related ruleset in nftables:
table ip libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output }
chain guest_output { ip saddr 192.168.121.0/24 iif "virbr1" counter packets 0 bytes 0 accept iif "virbr1" counter packets 0 bytes 0 reject }
chain guest_input { oif "virbr1" ip daddr 192.168.121.0/24 ct state established,related counter packets 0 bytes 0 accept oif "virbr1" counter packets 0 bytes 0 reject }
chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept }
chain guest_nat { type nat hook postrouting priority srcnat; policy accept; ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter packets 1 bytes 187 return ip saddr 192.168.121.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 meta l4proto udp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade } } table ip6 libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output }
chain guest_output { iif "virbr1" counter packets 0 bytes 0 reject }
chain guest_input { oif "virbr1" counter packets 0 bytes 0 reject }
chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept }
chain guest_nat { type nat hook postrouting priority srcnat; policy accept; } }
Here's some log output (I have virtnetworkd running with -v):
$ sudo journalctl -u virtnetworkd.service | grep -i nft [snip] Feb 20 21:36:42 stodi.digitalkingdom.org virtnetworkd[89119]: using firewall_backend: 'nftables' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip6 libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output ip saddr 192.168.121.0/24 iif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 ip daddr 192.168.121.0/24 ct state related,established counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto udp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 255.255.255.255/32 counter return' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter return'
So it's not like trying to create the rules and failing, that I can see; it just isn't trying.
To clarify this setup appears to be missing every rule that would be needed for DHCPv4 and DNS to work.
Like https://gitlab.com/libvirt/libvirt/-/issues/88#note_694493261 shows rules for udp 67 and 53 that are just 100% not being created at all.
I have no idea what's going wrong here or even where to look; the code at
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/GT75X... sure *looks like* it should be unconditionally adding those rules.
Help?
*sigh* Found this is the code: #ifdef VIR_NFTABLES_INCLUDE_HOST_RULES /* These rules copied from the iptables backend, have been removed * from the nftab because they are redundant since we are using our own * table that is default accept; there are no other users that * could add a reject rule that we would need to / be able to * override with these rules */ /* allow DHCP requests through to dnsmasq & back out */ nftablesAddTcpInput(fw, VIR_FIREWALL_LAYER_IPV4, def->bridge, 67); nftablesAddUdpInput(fw, VIR_FIREWALL_LAYER_IPV4, def->bridge, 67); nftablesAddTcpOutput(fw, VIR_FIREWALL_LAYER_IPV4, def->bridge, 68); nftablesAddUdpOutput(fw, VIR_FIREWALL_LAYER_IPV4, def->bridge, 68); Turns out this is correct, so it *was* an nftables ordering problem, I just didn't know that libvirt didn't lay those rules down in nftables mode so it looked broken. Once I removed the relevant reject in my nftables config, it worked. On Fri, Feb 21, 2025 at 12:06 AM Robin Powell <robinleepowell@gmail.com> wrote:
Setting `firewall_backend = "iptables"` and allowing it to use the iptables-nft shim *did* get the right rules into place.
I then ran smack into the "my main input chain has a default deny rule", but that's my problem. :)
On Thu, Feb 20, 2025 at 10:33 PM <robinleepowell@gmail.com> wrote:
I do not think this is any of the similar issues that have been posted to this list; I've checked.
In particular, this in *NOT* an issue with nft rules precedence; the rules are simply not being written by libvirt.
I'm running vagrant which is running libvirt on a Fedora 41 host. I do not think this is a vagrant problem, but if people want me to run virsh commands directly I certainly can.
Anyway, vagrant was failing until I noticed that the default for /etc/libvirt/network.conf is now nftables, which I did not have set up. When I set `firewall_backend = "iptables"`, everything worked fine.
I want to emphasize that: the same vagrant / libvirt setup *was* working with iptables.
But I took that as a sign that it was time to move to nftables, so I moved everything on this host and stuff is back to working.
But libvirt just isn't writing out the right nft rules. Like, at all.
Here's the network vagrant creates:
$ sudo virsh net-dumpxml vagrant-libvirt <network connections='1' ipv6='yes'> <name>vagrant-libvirt</name> <uuid>b2d93ef4-b305-4382-a380-c1eca92d8ebd</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:63:d3:f6'/> <ip address='192.168.121.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.121.1' end='192.168.121.254'/> </dhcp> </ip> </network>
And here's the *entire* libvirt-related ruleset in nftables:
table ip libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output }
chain guest_output { ip saddr 192.168.121.0/24 iif "virbr1" counter packets 0 bytes 0 accept iif "virbr1" counter packets 0 bytes 0 reject }
chain guest_input { oif "virbr1" ip daddr 192.168.121.0/24 ct state established,related counter packets 0 bytes 0 accept oif "virbr1" counter packets 0 bytes 0 reject }
chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept }
chain guest_nat { type nat hook postrouting priority srcnat; policy accept; ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter packets 1 bytes 187 return ip saddr 192.168.121.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 meta l4proto udp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade } } table ip6 libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output }
chain guest_output { iif "virbr1" counter packets 0 bytes 0 reject }
chain guest_input { oif "virbr1" counter packets 0 bytes 0 reject }
chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept }
chain guest_nat { type nat hook postrouting priority srcnat; policy accept; } }
Here's some log output (I have virtnetworkd running with -v):
$ sudo journalctl -u virtnetworkd.service | grep -i nft [snip] Feb 20 21:36:42 stodi.digitalkingdom.org virtnetworkd[89119]: using firewall_backend: 'nftables' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip6 libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output ip saddr 192.168.121.0/24 iif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 ip daddr 192.168.121.0/24 ct state related,established counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto udp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 255.255.255.255/32 counter return' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter return'
So it's not like trying to create the rules and failing, that I can see; it just isn't trying.
To clarify this setup appears to be missing every rule that would be needed for DHCPv4 and DNS to work.
Like https://gitlab.com/libvirt/libvirt/-/issues/88#note_694493261 shows rules for udp 67 and 53 that are just 100% not being created at all.
I have no idea what's going wrong here or even where to look; the code at
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/GT75X... sure *looks like* it should be unconditionally adding those rules.
Help?
participants (2)
-
Robin Powell -
robinleepowell＠gmail.com