Re: [libvirt] Reproducible VM start bug which affects libvirt 5.1.0 and 5.2.0

> > To fix this issue I have to restart libvirt. Some iptable chains are missing, which is probably caused by a nwfilter-define operation. > > I'm able to reproduce this bug within 2 hours by running 2 loops. One loop is defining nwfilters and the second loop is destroying and starting multiple VMs. > The fact that we recreate it everytime we try to start a guest also > means any problem should be self-correcting which makes it even more > strange that you need to have a restart It seems that the problem is a race condition between libvirt and our reload-iptables script. Libvirt inserts and removes rules one by one, while reload-iptables uses iptables-save and iptables-restore. The script reload-iptables saves libvirt firewall rules to a temp-file appends puppet's rules, and then imports said temp-file. When libvirt is inserting firewall rules between the save and import from reload-iptables we get unexpected behaviour.

> > So far I am not able to reproduce this bug on libvirt 5.0.0. > This is interesting, because AFAICT we had no changes to the nwfilter > driver between 5.0.0 and 5.1.0 that would affect this behaviour. > We did have the changes to the virtual network driver but that should not interfere with the nwfilter driver. The hypervisor running libvirt 5.0.0 was not using this reload-iptables script.