- Users - libvirt List Archives

Clarifying isolated network rules in nftables
by Alexey Kashavkin 27 Mar '25

27 Mar '25

Hi, I’m trying to understand how firewall filter works for isolated network in libvirt v11.1.0. When I start the network I can see following rules in nftables: table ip libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output } chain guest_output { iif "virbr3" counter packets 0 bytes 0 reject } chain guest_input { oif "virbr3" counter packets 0 bytes 0 reject } chain guest_cross { iif "virbr3" oif "virbr3" counter packets 0 bytes 0 accept } chain guest_nat { type nat hook postrouting priority srcnat; policy accept; } } But when I start ping from one VM to another on the same isolated network, I don't see an increase in counters in either chain. In the libvirt code, I found a comment in src/network/network_nftables.c: /** * nftablesAddForwardAllowCross: * * Add a rule to @fw to allow traffic to go across @iface (the virtual * network's bridge) from one port to another. This allows all traffic * between guests on the same virtual network. */ But it seems that these rules don't work and are not needed. If I delete this table or some chains, nothing happens. VMs have connectivity with each other on this network. What are these rules for?

2 1

Capture pcap for each VM
by icefrog1950＠gmail.com 26 Mar '25

26 Mar '25

Hi, Is possible to capture pcaps for each VM individually? QEMU supports command line '-object filter-dump, file=test.pcap'. I'm not sure if Libvirt supports this features, or there are better ways to solve this. Many thanks.

2 1

Running DHCP server inside of the Guest VM instead of the host.
by gameplayer2019pl＠tutamail.com 26 Mar '25

26 Mar '25

Hello, I've recently tried to run KEA DHCP server inside of a Debian VM and the following host interface configuration, attached to that VM: ``` <network> <name>netasn-dhcpv6</name> <bridge name="netasn-dhcpv6" stp="on" delay="0"/> <mtu size="1500"/> <mac address="XX:XX:XX:XX:XX:XX"/> <dns enable="no"/> <ip family="ipv6" address="2a14:7581:feb::1" prefix="64"> </ip> </network> ``` But whenever I'm trying to run dhclient on another Debian VM attached to the same network I couldn't obtain the IPv6 lease from KEA DHCP. The host is as well running Debian, 12 version. Is there anyway to make the DHCP work from Guest VM instead of using built-in libvirt DHCP?

2 2

Integrating Open Virtual Network with Libvert Provisioned VMs
by odukoyaonline＠gmail.com 26 Mar '25

26 Mar '25

I am having a little confusion trying to integrate OVN defined networks(I tried with Ovs and it worked fine) with the VMs I have provisioned with Libvirt. I want to ask if there are resource around this and if not, will appreciate input from anyone who has a prior experience with it no matter how little.

1 0

General use of zstd instead of zlib compression
by Michael Niehren 17 Mar '25

17 Mar '25

Hi, currently i use on all VM's qcow2-Images with zlib compression. If i do an Backup, the Backup-Image will be compressed with zstd Level 3 to shrink the image for transfering it over not so fast internet connections. So, why not directly using zstd compression on the images. Are there any reason's not to do that ? As i always use virt-manager for administration, i patched qemu (V9.2.2) to create on default zstd compressed images (only 1 change in line 3525). So newly created images do have compression type zstd, which work's (qemi-img info). I see one unusual thing. If i do an qemi-img convert with compression_type=zstd the size of the converted image is much smaller than the original file while "qemu-img info" shows on both as compression type zstd. Do they use different compression levels, maybe ? If i now do an virsh backup-begin <domain>, the backup-Image does also has a bigger size than the original, while showing zstd as compression type (qemu-img info). If i convert it with the similar command as above, both converted images has nearly the same size. Even if i copy the smaller converted image to the original and boot the vm from the smaller image, the backup-image (after backup-begin) is bigger. So, i am confused. Are there any explanations about the different image sizes or what's going on here ? best regards Michael

3 10

live migration of SR-IOV vm
by Paul B. Henson 17 Mar '25

17 Mar '25

I have a vm using an sr-iov NIC that I'm testing live migration on (Debian 12, OS packages). Per the documentation, I have the sr-iov link set as transient with a pointer to the persistent virtio link: <interface type='network'> <mac address='52:54:00:a1:e0:38'/> <source network='sr-iov-intel-10G-1'/> <vlan> <tag id='400'/> </vlan> <model type='virtio'/> <teaming type='transient' persistent='ua-sr-iov-backup'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> and a persistent virto link down by default: <interface type='direct'> <mac address='52:54:00:a1:e0:38'/> <source dev='eno5np0.400' mode='bridge'/> <model type='virtio'/> <teaming type='persistent'/> <link state='down'/> <alias name='ua-sr-iov-backup'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </interface> The failover driver finds this in the vm: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 100 0 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master eth0 state DOWN mode DEFAULT group default qlen 1000 10: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master eth0 state UP mode DEFAULT group default qlen 1000 and the network works fine. However, during migration, the sr-iov interface is removed, but the link on the virtio interface is *not* brought up: 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master eth0 state DOWN mode DEFAULT group default qlen 1000 resulting in no network for part of the migration. Once the box finished migrating, the replacement sv-iov link is plugged back in, and all is well once again: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master eth0 state DOWN mode DEFAULT group default qlen 1000 11: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master eth0 state UP mode DEFAULT group default qlen 1000 My understanding was that virsh migrate was supposed to automatically bring up the virtio link when the sr-iov link is removed? Or do I need to explicitly bring it up myself before the migration and take it down after? If I bring up the link manually before the migration, there are a few packets lost at that time, but then none are lost during the migration when the sr-iov link is pulled, or after the migration when I shut that link down again. Ideally no packets would be lost :), but I realize that's unlikely in practice... Thanks...

1 1

system cache dropped
by matus valentin 14 Mar '25

14 Mar '25

Hi, I have a setup with multiple virtual machines (VMs), each with a saved state. All VMs share the same parent, which is located on a shared drive. Whenever I restore any VM using virsh restore, the parent is dropped from the system cache, causing all data to be downloaded from the shared drive again. This results in unnecessary network traffic, even though the parent changes very rarely. However, if I create a child from the parent and call virsh create to create a new VM, the parent is never dropped from the system cache. Is this expected behavior? Should the parent be retained in the system cache during a virsh restore operation? Are there any configurations or settings that can prevent the parent from being dropped from the cache? thanks

1 0

best backup strategy for full backup's
by Michael Niehren 10 Mar '25

10 Mar '25

Hi together, actually i only do full-backup's of my virtual machines. I use the for the backup the "old" strategy: - virsh snapshot-create-as $vmname overlay --disk-only --atomic --no-metadata --quiesce - copy the qcow2 image file - virsh blockcommit $vmname $device --active --wait --pivot - the guest agent in the VM got's an 2 seconds freeze/thaw intervall Now i want to switch to the new strategy with "backup-begin". - virsh backup-begin $vmname - the guest agent does not got an freez/thaw signal As the guest agent got's no signal, is the backup over "backup-begin" still consistent ? Or do i have to be consistent to send an virsh domfsfreeze $vmname before starting the backup and an virsh domfsthaw $vmname it it is finished ? If so, the time intervall between freeze/thaw would be on an huge disk much more then 2 secords. So, is the old method currently still the better way, if only doing full-backup's ? best regards, Michael

2 6

So hey let's talk about this nftables ordering situation.
by robinleepowell＠gmail.com 25 Feb '25

25 Feb '25

So I, like many other people, have hit problems with nftables ordering, as has been discussed on this mailing list MANY TIMES. This whole thing seemed ridiculous so I asked the nftables people about what one is *supposed* to do in this situation. It turns out that the standard solution is for libvirt's nftables rules to set a packet mark (there's a collision possibility here but it's a 32 bit integer if you pick one at random it shouldn't be a problem) and then the user adds a rule to exclude packets with that mark from any reject rules they might have, or explicitly accept marked packets in their own chains, or whatever. It's not *as nice* as the iptables situation, but having documentation that says "if you're using nftables make sure that packets with mark 79892 are accepted in all your chains" is quite straightforward compared to the current situation of "LOL good luck". (I'm not blaming anyone there!, the current situation is impossible for libvirt to navigate and it's not anyone's fault.) If y'all don't like that, what's working excellently for me is adding `iifname "virbr*" accept` to my rule chain. FWIW. It was very hard to navigate through this situation because there's no documentation that this problem even exists. My suggestion is to describe the situation at https://libvirt.org/firewall.html and suggest the virbr* fix, and down the road maybe look at this mark thing. I'd like to help. I'm happy to write up issues for this, and I'm happy to write the updates to the firewall docs; just tell me what you'd like me to do.

4 8

nftables rules (DNS, DHCP, etc) not being written on Fedora 41
by robinleepowell＠gmail.com 21 Feb '25

21 Feb '25

I do not think this is any of the similar issues that have been posted to this list; I've checked. In particular, this in *NOT* an issue with nft rules precedence; the rules are simply not being written by libvirt. I'm running vagrant which is running libvirt on a Fedora 41 host. I do not think this is a vagrant problem, but if people want me to run virsh commands directly I certainly can. Anyway, vagrant was failing until I noticed that the default for /etc/libvirt/network.conf is now nftables, which I did not have set up. When I set `firewall_backend = "iptables"`, everything worked fine. I want to emphasize that: the same vagrant / libvirt setup *was* working with iptables. But I took that as a sign that it was time to move to nftables, so I moved everything on this host and stuff is back to working. But libvirt just isn't writing out the right nft rules. Like, at all. Here's the network vagrant creates: $ sudo virsh net-dumpxml vagrant-libvirt <network connections='1' ipv6='yes'> <name>vagrant-libvirt</name> <uuid>b2d93ef4-b305-4382-a380-c1eca92d8ebd</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:63:d3:f6'/> <ip address='192.168.121.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.121.1' end='192.168.121.254'/> </dhcp> </ip> </network> And here's the *entire* libvirt-related ruleset in nftables: table ip libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output } chain guest_output { ip saddr 192.168.121.0/24 iif "virbr1" counter packets 0 bytes 0 accept iif "virbr1" counter packets 0 bytes 0 reject } chain guest_input { oif "virbr1" ip daddr 192.168.121.0/24 ct state established,related counter packets 0 bytes 0 accept oif "virbr1" counter packets 0 bytes 0 reject } chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept } chain guest_nat { type nat hook postrouting priority srcnat; policy accept; ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter packets 1 bytes 187 return ip saddr 192.168.121.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 meta l4proto udp ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 ip saddr 192.168.121.0/24 ip daddr != 192.168.121.0/24 counter packets 0 bytes 0 masquerade } } table ip6 libvirt_network { chain forward { type filter hook forward priority filter; policy accept; counter packets 0 bytes 0 jump guest_cross counter packets 0 bytes 0 jump guest_input counter packets 0 bytes 0 jump guest_output } chain guest_output { iif "virbr1" counter packets 0 bytes 0 reject } chain guest_input { oif "virbr1" counter packets 0 bytes 0 reject } chain guest_cross { iif "virbr1" oif "virbr1" counter packets 0 bytes 0 accept } chain guest_nat { type nat hook postrouting priority srcnat; policy accept; } } Here's some log output (I have virtnetworkd running with -v): $ sudo journalctl -u virtnetworkd.service | grep -i nft [snip] Feb 20 21:36:42 stodi.digitalkingdom.org virtnetworkd[89119]: using firewall_backend: 'nftables' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft list table ip6 libvirt_network' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_output iif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_input oif virbr1 counter reject' Feb 20 21:36:46 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip6 libvirt_network guest_cross iif virbr1 oif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_output ip saddr 192.168.121.0/24 iif virbr1 counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_input oif virbr1 ip daddr 192.168.121.0/24 ct state related,established counter accept' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto udp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat meta l4proto tcp ip saddr 192.168.121.0/24 ip daddr '!=' 192.168.121.0/24 counter masquerade to :1024-65535' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 255.255.255.255/32 counter return' Feb 20 21:36:47 stodi.digitalkingdom.org virtnetworkd[89119]: Applying 'nft -ae insert rule ip libvirt_network guest_nat ip saddr 192.168.121.0/24 ip daddr 224.0.0.0/24 counter return' So it's not like trying to create the rules and failing, that I can see; it just isn't trying. To clarify this setup appears to be missing every rule that would be needed for DHCPv4 and DNS to work. Like https://gitlab.com/libvirt/libvirt/-/issues/88#note_694493261 shows rules for udp 67 and 53 that are just 100% not being created at all. I have no idea what's going wrong here or even where to look; the code at https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/GT75… sure *looks like* it should be unconditionally adding those rules. Help?

2 2