Re: [libvirt] Reproducible VM start bug which affects libvirt 5.1.0 and 5.2.0
On Wed, Apr 03, 2019 at 10:52:57AM +0000, Frank Schreuder wrote:
Hello,
I am currently running into a reproducible libvirt bug which affects libvirt 5.1.0 and
5.2.0.
There seem to be a racecondition in the nwfilter-define and virsh start
commands. Several times a day I'm not able to start a VM anymore with
the following error message:
error: Failed to start domain test
error: internal error: Failed to apply firewall rules /sbin/iptables -w -I FORWARD 1 -j
libvirt-in: iptables v1.6.0: Couldn't load target `libvirt-in':No such file or
directory
"libvirt-in" is a chain created by the nwfilter driver to hold its rules
AFAICT, it tries to re-create this chain every time. The code runs this
series of ops:
virFirewallAddRuleFull(fw, layer,
true, NULL, NULL,
"-N", VIRT_IN_CHAIN, NULL);
virFirewallAddRuleFull(fw, layer,
true, NULL, NULL,
"-D", "FORWARD", "-j",
VIRT_IN_CHAIN, NULL);
virFirewallAddRule(fw, layer,
"-I", "FORWARD", "1", "-j",
VIRT_IN_CHAIN, NULL);
where "VIRT_IN_CHAIN" is a macro expanding to "libvirt-in". So AFAICT
it should be impossible to hit the problem you are showing, since we
just created the "libvirt-in" chain before trying to add it to the
FORWARD chain !
To fix this issue I have to restart libvirt. Some iptable chains are
missing, which is probably caused by a nwfilter-define operation.
I'm able to reproduce this bug within 2 hours by running 2 loops. One loop is
defining nwfilters and the second loop is destroying and starting multiple VMs.
The fact that we recreate it everytime we try to start a guest also
means any problem should be self-correcting which makes it even more
strange that you need to have a restart
I found an entry in the changelog of libvirt 5.1.0, which seems
related to this bug:
Create private chains for virtual network firewall rules
Historically firewall rules for virtual networks were added straight into the base
chains. This works but has a number of bugs and design limitations. To address them,
libvirt now puts firewall rules into its own chains. Note that with this change the
filter, nat and mangle tables are required for both IPv4 and IPv6.
This is relating tot he virtual network driver's firewall rules, which
are completely independnat of the nwfilter driver's firewall rules.
This changelog entry is refering to the new top level chains we create:
INPUT --> LIBVIRT_INP (filter)
OUTPUT --> LIBVIRT_OUT (filter)
FORWARD +-> LIBVIRT_FWX (filter)
+-> LIBVIRT_FWO
\-> LIBVIRT_FWI
POSTROUTING --> LIBVIRT_PRT (nat & mangle)
as you can see these are different chains from the nwfilter ones
you mention.
So far I am not able to reproduce this bug on libvirt 5.0.0.
This is interesting, because AFAICT we had no changes to the nwfilter
driver between 5.0.0 and 5.1.0 that would affect this behaviour.
We did have the changes to the virtual network driver but that should
not interfere with the nwfilter driver.
Is there any information I can provide to the mailinglist to help
debug and/or fix this bug? I am also willing to test patches.
Probably only thing is to turn on debugging
log_filters="1:libvirt 1:firewall 1:iptables 1:nwfilter 1:network"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
in libvirtd.conf & restart libvirtd.
Since you say this takes 2 hours to reprorduce though I think this is going
to create a *HUGE* logfile. We'll probably only need the last few 100 KB
or so, of the logfile.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|