Re: [libvirt] [RFC] Introduce API for retrieving bulk domain stats

On Tue, Aug 19, 2014 at 03:14:19PM +0200, Peter Krempa wrote: > I'd like to propose a (hopefully) fairly future-proof API to retrieve > various statistics for domains. > > The motivation is that management layers that use libvirt usually poll > libvirt for statistics using various split up APIs we currently provide. > To get all the necessary stuff, the mgmt app need to issue Ndomains * > Napis calls and cope with the various returned formats. The APIs I'm > wanting to introduce here will: > > 1) Return data in a format that we can expand in the future and is > hierarchical. For starters I'll use XML, with possible expansion to > something like JSON if it will be favourable for a consumer (switchable > by a flag) I'm not particularly a fan of using XML for this. As a guiding principal we've used structures when we needed APIs for which efficiency is important, and XML for APIs where efficiency is irrelevant. Even with the ability to bulk list data from many VMs at once, I'd think that efficiency is still a very important property of the API. Consider if we have 1000 virtual machines on a host - the XML document is going to get very large and frankly most XML parsers are terribly slow. Would not surprise me if it too several seconds or more to parse an XML doc that this proposed API would return, which I don't think is really viable. JSON might be slightly better in this respect, but not by as much as we might imagine. So I'd rather think we need to figure out a way to map this into some kind of struct, where reading any single statistic could be done in approx constant time, or at least time that is independant of the number of VMs. Much as I dislike the virTypedParameter struct, it might actually be a reasonable fit here. Perhaps a struct struct virDomainRecord { virDomainPtr dom; size_t nparams; virTypedParameter params; }; and the API returns an array of virDomainRecord elements. For the keys in the parameters we could use dot separated components. eg state=runing cpu.online=8 cpu.0.state=running cpu.0.time=10001231231 cpu.1.state=running cpu.1.time=10001231231 ... cpu.7.state=running cpu.7.time=10001231231 This can be fairly efficiently accessed without any parsing involved.

The app would have todo an O(n) scan over the array to record the mapping of UUID -> array indexes. Once that's done any single stat can be accessed in O(1) time. I don't think anything XML or JSON based could even come close to this kind of efficiency of access, not to mention that it avoids the need for apps to write XML parsers which will simplify their life no end.