Re: [libvirt] [PATCH 0/4] improve virConnectListAllInterfaces()

On Fri, Sep 25, 2015 at 11:13:52AM -0400, Laine Stump wrote: > There's a bit of background about this here: > > https://www.redhat.com/archives/augeas-devel/2015-September/msg00001.html > > In short, virt-manager is calling the virInterface APIs and that ties > up a libvirt thread (and CPU core) for a very long time on hosts that > have a large number of interfaces. These patches don't cure the > problem (I don't know that there really is a cure other than "Don't DO > that!"), but they do fix a couple of bugs I found while investigating, > and make a substantial improvement in the amount of time used by > virConnectListAllInterfaces(). > > One thing that I wondered about while investigating this - a big use > of CPU by virConnectListAllInterfaces() comes from the need to > retrieve the MAC address of every interface. The MAC addresses are > both > > 1) returned to the caller in the interface objects and > > 2) sent to the policykit ACL checking to decide which interfaces to include in > the list. > > I'm 90% confident that > > 1) most callers don't really care that they're getting the MAC address > along with interface name (virt-manager, for example, follows up with > a virInterfaceGetXMLDesc() anyway)), and > > 2) there is not even a single host *anywhere* that is using libvirt > policykit ACLs to limit the list of host interfaces viewable by a > client. > > So we could add a flag to not return MAC addresses, which would allow > cutting down the time to list all devices to something < 1 > second). But this would be at the expense of no longer having the > possibility to limit the list with policykit according to MAC > address. Since all host interface information is available to all > users via the file system, for example, I don't see this as a huge > issue, but it would change behavior, so I don't feel comfortable doing > it. I don't like the idea of a single API call taking > 1 minute to > return either, though. Does anyone have an opinion about this? Ultimately 500 interfaces, each ifcfg-XXX file 300 bytes in size on average is only 150 KB of data. Given the amount of data we are consuming, here I think it is reasonable to expect we can process than in a tiny fraction of a second. So there's clearly a serious algorithmic / data structure flaw here if its taking minutes. By the sounds of the thread you quote, its in augeas itself, so I think we really need to focus on addressing that root cause as a priority rather than try to work around it. As a side note, we might consider adding new API to netcf so that we can fetch the entire interface set + macs in one api call to netcf, though I doubt it'd chance performance that much.