Re: [libvirt] [RFC] Faster libvirtd restart with nwfilter rules

Hi, all. On fat hosts which are capable to run hundreds of VMs restarting libvirtd makes it's services unavailable for a long time if VMs use network filters. In my tests each of 100 VMs has no-promisc [1] and no-mac-spoofing filters and executing virsh list right after daemon restart takes appoximately 140s if no firewalld is running (that is ebtables/iptables/ip6tables commands are used to configure kernel tables). The problem is daemon does not even start to read from client connections because state drivers are not initialized. Initialization is blocked in state drivers autostart which grabs VMs locks. And VMs locks are hold by VMs reconnection code. Each VM reloads network tables on reconnection and this reloading is serialized on updateMutex in gentech nwfilter driver. Workarounding autostart won't help much because even if state drivers will initialize listing VM won't be possible because listing VMs takes each VM lock one by one too. However managing VM that passed reconnection phase will be possible which takes same 140s in worst case. Note that this issue is only applicable if we use filters configuration that don't need ip learning. In the latter case situation is different because reconnection code spawns new thread that apply network rules only after ip is learned from traffic and this thread does not grab VM lock. As result VMs are managable but reloading filters in background takes appoximately those same 140s. I guess managing network filters during this period can have issues too. Anyway this situation does not look good so fixing the described issue by spawning threads even without ip learning does not look nice to me. What speed up is possible on conservative approach? First we can remove for test purpuses firewall ruleLock, gentech dirver updateMutex and filter object mutex which do not serve function in restart scenario. This gives 36s restart time. The speed up is archived because heavy fork/preexec steps are now run concurrently. Next we can try to reduce fork/preexec time. To estimate its contibution alone let's bring back the above locks. It turns out the most time takes fork itself and closing 8k (on my system) file descriptors in preexec. Using vfork gives 2x boost and so does dropping mass close. (I check this mass close contribution because I not quite understand the purpose of this step - libvirt typically set close-on-exec flag on it's descriptors). So this two optimizations alone can result in restart time of 30s. Unfortunately combining the above two approaches does not give boost multiple of them along. The reason is due to concurrency and high number of VMs (100) preexec boost does not have significant role and using vfork dininishes concurrency as it freezes all parent threads before execve. So dropping locks and closes gives 33s restart time and adding vfork to this gives 25s restart time. Another approach is to use --atomic-file option for ebtables (iptables/ip6tables unfortunately does not have one). The idea is to save table to file/edit file/commit table to kernel. I hoped this could give performance boost because we don't need to load/store kernel network table for a single rule update. In order to isolate approaches I also dropped all ip/ip6 updates which can not be done this way. In this approach we can not drop ruleLock in firewall because no other VM threads should change tables between save/commit. This approach gives restart time 25s. But this approach is broken anyway as we can not be sure another application doesn't change newtork table between save/commit in which case these changes will be lost. After all I think we need to move in a different direction. We can add API to all binaries and firewalld to execute many commands in one run. We can pass commands as arguments or wrote them into file which is then given to binary. Then libvirt itself can update for example bridge network table in couple of commands. The exact number depends on new API. For example if we add option to delete chains recursively and an option not to fail on NOENT error we can change table in one command (no listing current rules is required). [1] no-promisc filter <filter name='no-promisc' chain='root' priority='-750'> <uuid>6d055022-1192-4a3d-ae1f-576baa5564b6</uuid> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='ff:ff:ff:ff:ff:ff'/> </rule> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='$MAC'/> </rule> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='33:33:00:00:00:00' dstmacmask='ff:ff:00:00:00:00'/> </rule> <rule action='drop' direction='in' priority='500'> <mac/> </rule> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='01:00:5e:00:00:00' dstmacmask='ff:ff:ff:80:00:00'/> </rule> </filter> -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list