Re: [libvirt] [RFC] Faster libvirtd restart with nwfilter rules

On Thu, Oct 11, 2018 at 03:57:10PM -0400, Laine Stump wrote: > On 10/11/2018 06:20 AM, Daniel P. Berrangé wrote: > > On Mon, Sep 24, 2018 at 10:41:37AM +0300, Nikolay Shirokovskiy wrote: > > > >> What speed up is possible on conservative approach? First we can remove for > >> test purpuses firewall ruleLock, gentech dirver updateMutex and filter object > >> mutex which do not serve function in restart scenario. This gives 36s restart > >> time. The speed up is archived because heavy fork/preexec steps are now run > >> concurrently. > > To me the most obvious speedup is not to run any commands at all. > > > > 99% of the time when we restart libvirtd and re-build network filters > > we are achieving nothing useful. We tear down the rules and replace > > them by exactly the same rules. > > > > The idea behind rebuilding at startup is that someone might have changed > > the config on diks while libvirtd was not running, and we want to ensure > > that the VMs have the correct live config after this. > > > > Alternatively libvirtd has been upgraded to a newer version, and we want > > to rebuild with the new code in case old code had a bug causing it to > > create incorrect rules. > > > > I think we should look at how to optimize this to be more intelligent. > > > > We could somehow record a hash of what rule contents was used > > originally. Only rebuild the rules if we see the hash of nwfilter > > rules has changed, or if libvirtd binary has had code changes. > > Since we pre-emptively delete any existing copy of a rule just prior to > adding that rule, another way we could reduce startup time would be to > spend less time deleting. one way of doing that would (I think) be if we > could add all of our rules in a set of chains that we create ourselves. > Then when libvirtd restarted, we could just issue a few commands to > flush those chains. (come to think of it, nwfilter *already* puts > most/all of its rules in self-created chains. Hmm.... I don't think deletion is a big problem - we only delete the custom chains with nwfilter, and that's a small part of the command count. > The best way to reduce fork/exec time is of course to never fork/exec at > all :-). So, a chain of current/future events that could lead to 0 > fork/execs: > > 1) upstream firewalld now has an nftables backend available >    (hang with me, this really is related...) > > 2) some distros have already switched to it (someone running debian on > #virt a couple weeks ago had a problem that turned out to be due to > using the nftables backend, and Fedora 29 *tried* to switch to using the > nftables backend, but had to revert/postpone the change due to the > networking problems it caused with libvirt virtual networks.) > > 3) due to (1) and (2) we're looking into making libvirt work properly > with a firewalld using nftables (it's going to take new code in both > libvirt *and* firewalld) > > 4) an upcoming release of firewalld will be to adding/deleting nftables > rules using calls to a library rather than a fork/exec of the nft command. > > Once libvirt is working with an nftables backend to firewalld, and > firewalld is using an API to access nftables, there will be 0 > fork/execs! Yay! Sadly not. We don't fork/exec but we do call dbus *synchronously* to get firewalld to add the rules, and firewalld does a fork/exec. IOW, we still have the fork/exec for every command, but we also now have a dbus roundtrip on top.