[Libvir] PATCH: Improve performance on Xen
The Xen implementations of virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece & have to lookup the other piece. Every running domain has to have an entry in XenStore on /local/domain/N where 'N' is the id. Under this location there is always a 'name' field. So if we have the id we can efficiently get the name. I just realized that the getdomaininfo hypercall struct contains the uuid and id. So for the ByID and ByUUID calls we can get all the info we need with a hypercall and a read of xenstore. This is very efficient compared to hitting XenD. As of Xen 3.1.0 hypervisor the flags in the getdomaininfo hypercall struct also mark whether the guest is HVM vs paraivrt, so we can also implement the GetOSType api with a single hypercall. That just leaves the ByName impl. The only way I can think of doing this is to scan /local/domain/N in xenstore until we find it. I'm going to try this, but I'm not convinced it'll be any significantly than talking to XenD. I may be surprised though. Any for the 3 improvements I have done, the 'virsh dominfo' call has improved when using either ID or UUID: Create a set of UUIDs & IDs # for i in `seq 1 500` ; do echo dominfo 73 ; done > ids.txt # for i in `seq 1 500` ; do echo dominfo 8f07fe28-753f-2729-d76d-bdbd892f949a ; done > uuids.txt Now before measurements: # time ./virsh < ids.txt > /dev/null real 0m17.317s user 0m0.144s sys 0m0.164s # time ./virsh < uuids.txt > /dev/null real 0m14.432s user 0m0.236s sys 0m0.460s Against the after measurements: # time ./virsh < demo.txt > /dev/null real 0m0.259s user 0m0.100s sys 0m0.072s # time ./virsh < demo.txt > /dev/null real 0m0.482s user 0m0.324s sys 0m0.108s A pretty good speedup really considering how frequently these calls are made if you're monitoring domains frequently. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Daniel P. Berrange wrote:
The Xen implementations of
virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType
all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece & have to lookup the other piece.
Every running domain has to have an entry in XenStore on /local/domain/N where 'N' is the id. Under this location there is always a 'name' field. So if we have the id we can efficiently get the name. I just realized that the getdomaininfo hypercall struct contains the uuid and id. So for the ByID and ByUUID calls we can get all the info we need with a hypercall and a read of xenstore. This is very efficient compared to hitting XenD.
As of Xen 3.1.0 hypervisor the flags in the getdomaininfo hypercall struct also mark whether the guest is HVM vs paraivrt, so we can also implement the GetOSType api with a single hypercall.
That just leaves the ByName impl. The only way I can think of doing this is to scan /local/domain/N in xenstore until we find it. I'm going to try this, but I'm not convinced it'll be any significantly than talking to XenD. I may be surprised though.
Any for the 3 improvements I have done, the 'virsh dominfo' call has improved when using either ID or UUID:
Create a set of UUIDs & IDs
# for i in `seq 1 500` ; do echo dominfo 73 ; done > ids.txt # for i in `seq 1 500` ; do echo dominfo 8f07fe28-753f-2729-d76d-bdbd892f949a ; done > uuids.txt
Now before measurements:
# time ./virsh < ids.txt > /dev/null real 0m17.317s user 0m0.144s sys 0m0.164s
# time ./virsh < uuids.txt > /dev/null real 0m14.432s user 0m0.236s sys 0m0.460s
Against the after measurements:
# time ./virsh < demo.txt > /dev/null real 0m0.259s user 0m0.100s sys 0m0.072s
# time ./virsh < demo.txt > /dev/null real 0m0.482s user 0m0.324s sys 0m0.108s
A pretty good speedup really considering how frequently these calls are made if you're monitoring domains frequently.
+1 Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903
On Fri, Aug 10, 2007 at 10:57:50AM +0100, Richard W.M. Jones wrote:
Daniel P. Berrange wrote:
The Xen implementations of
virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType
all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece & have to lookup the other piece.
A pretty good speedup really considering how frequently these calls are made if you're monitoring domains frequently.
+1
Rich.
Ok, just commited this now. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Thu, Aug 09, 2007 at 11:13:00PM +0100, Daniel P. Berrange wrote:
The Xen implementations of
virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType
all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece & have to lookup the other piece.
Every running domain has to have an entry in XenStore on /local/domain/N where 'N' is the id. Under this location there is always a 'name' field. So if we have the id we can efficiently get the name. I just realized that the getdomaininfo hypercall struct contains the uuid and id. So for the ByID and ByUUID calls we can get all the info we need with a hypercall and a read of xenstore. This is very efficient compared to hitting XenD.
Great !!!!
As of Xen 3.1.0 hypervisor the flags in the getdomaininfo hypercall struct also mark whether the guest is HVM vs paraivrt, so we can also implement the GetOSType api with a single hypercall.
Sounds good to though definitely not a bottleneck.
That just leaves the ByName impl. The only way I can think of doing this is to scan /local/domain/N in xenstore until we find it. I'm going to try this, but I'm not convinced it'll be any significantly than talking to XenD. I may be surprised though.
Well if you have 100 guests, that may be slower, but in the average situation of only a couple of guests, it could be a real speedup. The problem is that a lot of domain may accumulate in xenstore /local/domain even if they are not running, both implementation are likely to have completely different behaviour based on the context. But from a cache locality perspective hitting xenstore may scale way better under loaded machines, so it may prove faster even on machines with hundreds of domains. Doing a fair performance comparison may prove really hard.
Any for the 3 improvements I have done, the 'virsh dominfo' call has improved when using either ID or UUID:
[...]
A pretty good speedup really considering how frequently these calls are made if you're monitoring domains frequently.
Too good to ignore, that looks even too good to be true :-)
Index: src/xen_internal.c
Patch looks good to me +1, fanxastic ! Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Fri, Aug 10, 2007 at 06:39:56AM -0400, Daniel Veillard wrote:
On Thu, Aug 09, 2007 at 11:13:00PM +0100, Daniel P. Berrange wrote:
The Xen implementations of
virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType
all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece & have to lookup the other piece.
Every running domain has to have an entry in XenStore on /local/domain/N where 'N' is the id. Under this location there is always a 'name' field. So if we have the id we can efficiently get the name. I just realized that the getdomaininfo hypercall struct contains the uuid and id. So for the ByID and ByUUID calls we can get all the info we need with a hypercall and a read of xenstore. This is very efficient compared to hitting XenD.
Great !!!!
As of Xen 3.1.0 hypervisor the flags in the getdomaininfo hypercall struct also mark whether the guest is HVM vs paraivrt, so we can also implement the GetOSType api with a single hypercall.
Sounds good to though definitely not a bottleneck.
That just leaves the ByName impl. The only way I can think of doing this is to scan /local/domain/N in xenstore until we find it. I'm going to try this, but I'm not convinced it'll be any significantly than talking to XenD. I may be surprised though.
Well if you have 100 guests, that may be slower, but in the average situation of only a couple of guests, it could be a real speedup. The problem is that a lot of domain may accumulate in xenstore /local/domain even if they are not running, both implementation are likely to have completely different behaviour based on the context. But from a cache locality perspective hitting xenstore may scale way better under loaded machines, so it may prove faster even on machines with hundreds of domains. Doing a fair performance comparison may prove really hard.
Actually the /local/domain/[ID] subtree is guarenteed to only contain running VMs. The /vm/[UUID] subtree is where cruft accumulates over time, so its safe to rely on info in the former, but not the latter Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, Aug 10, 2007 at 02:33:28PM +0100, Daniel P. Berrange wrote:
On Fri, Aug 10, 2007 at 06:39:56AM -0400, Daniel Veillard wrote:
Well if you have 100 guests, that may be slower, but in the average situation of only a couple of guests, it could be a real speedup. The problem is that a lot of domain may accumulate in xenstore /local/domain even if they are not running, both implementation are likely to have completely different behaviour based on the context. But from a cache locality perspective hitting xenstore may scale way better under loaded machines, so it may prove faster even on machines with hundreds of domains. Doing a fair performance comparison may prove really hard.
Actually the /local/domain/[ID] subtree is guarenteed to only contain running VMs. The /vm/[UUID] subtree is where cruft accumulates over time, so its safe to rely on info in the former, but not the latter
Ah, right, I somehow remembered some nastyness there, but it wasn't precise :-) Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
participants (3)
-
Daniel P. Berrange -
Daniel Veillard -
Richard W.M. Jones