Re: [Libvir] Scability / performance fix for virDomainLookupByID

On Fri, Jul 07, 2006 at 03:47:59PM +0100, Daniel P. Berrange wrote: i So my previous post considered performance when doing a single iteration over all active domains. I've now moved onto look at what happens when you do multiple iterations - eg a monitoring application sampling state every 5 seconds. Take code which connects to the HV, and then periodically lists all domains and gets their status, eg conn = virConnectOpen(NULL); for (j = 0 ; j < 5 ; j++) { nid = virConnectListDomains(conn, ids, sizeof(ids)/sizeof(int)); for (i = 0 ; i < nid ; i++) { int id = ids[i]; dom = virDomainLookupByID(conn, id); ret = virDomainGetInfo(dom, &info); printf("Domains %d: %d CPUs\n", id, info.nrVirtCpu); } } Then, every single call to virDomainLookupByID will trigger an HTTP GET to resolve id -> name,uuid. Now the virConnect data structure contains a hash table storing all known domains, however, the xenDaemonLookupByID method only uses the cache once its done the HTTP GET to resolv id -> name. This is obviously needed the first time around, however, subsequent calls to xenDaemonLookupByID have no need to do the HTTP GET since the cache already contains a virDomain instance with id, name, uuid resolved. If you measure the above code snippet on a machine with 20 domains, then 5 iterations of the outer loop will require 101 HTTP GET's and take about 5 seconds to complete. If we modify the xenDaemonLookupByID method to check the cache immediately, doing a cache lookup on ID, rather than name, we only need to do the HTTP GET request the very first time a domain is looked up. Thus, only the first iteration of the outer loop will result in HTTP calls - you can do as many iterations as you like and there will only ever by 21 HTTP GETs performed, and total runtime for 5 iterations drops to 1 second. The first iteration does HTTP GETS to resolve name & uuid, but all subsquent iterations only need to do HyperCalls so are lightening fast. I'm attaching a patch which implements this speed up. The only issue not being dealt with here is cache eviction. Xen Domain IDs will always increase, no ID is re-used until the IDs wrap around - I can't remember whether the ID space is 16 or 32-bits, but even with 16bits we'd need 32,000 domains to have been created before wrap-around becomes an issue. We'd need to address it long term, but for now I think we can live with this. Cache evication could probaly be done in the virListDomains method - ie evict any entries with IDs not currently running Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|