[libvirt] MemoryPeek() is slow
Hi, I play around with MemoryPeek() API on QEMU. While it works well, I found that it is too slow. That is expected because of the way it works: we always need to save memory to a file, and read it in again, and that is too inefficient. I am trying to figure out a better way to do this. To do that, clearly we need to re-architecture QEMU for this: We must have another way to read memory from outside the QEMU process. Anybody could suggest a solution for this problem? I am willing to spend time on this feature to improve the situation. Thanks, J
On Tue, Jul 21, 2009 at 07:39:07PM +0900, Jun Koi wrote:
Hi,
I play around with MemoryPeek() API on QEMU. While it works well, I found that it is too slow.
Slow in what context ? Are you trying to read large amounts of data out of the guest ?
That is expected because of the way it works: we always need to save memory to a file, and read it in again, and that is too inefficient.
Possibly, but there's quite a few other variables in the stack that could impact performance too. eg you've got at least 3 more data copies in the libvirt RPC layer ontop of that. Also the libvirt RPC layer and API for memory peek has synchronous round-trips which will result in bad throughput if making lot sof peek calls in a row
I am trying to figure out a better way to do this. To do that, clearly we need to re-architecture QEMU for this: We must have another way to read memory from outside the QEMU process.
Anybody could suggest a solution for this problem? I am willing to spend time on this feature to improve the situation.
Could you explain in more detail what you are using the memory peek API for ? That might suggest a completely different libvirt API, or even something outside of libvirft Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Tue, Jul 21, 2009 at 7:50 PM, Daniel P. Berrange<berrange@redhat.com> wrote:
On Tue, Jul 21, 2009 at 07:39:07PM +0900, Jun Koi wrote:
Hi,
I play around with MemoryPeek() API on QEMU. While it works well, I found that it is too slow.
Slow in what context ? Are you trying to read large amounts of data out of the guest ?
Yes.
That is expected because of the way it works: we always need to save memory to a file, and read it in again, and that is too inefficient.
Possibly, but there's quite a few other variables in the stack that could impact performance too. eg you've got at least 3 more data copies in the libvirt RPC layer ontop of that. Also the libvirt RPC layer and API for memory peek has synchronous round-trips which will result in bad throughput if making lot sof peek calls in a row
I am trying to figure out a better way to do this. To do that, clearly we need to re-architecture QEMU for this: We must have another way to read memory from outside the QEMU process.
Anybody could suggest a solution for this problem? I am willing to spend time on this feature to improve the situation.
Could you explain in more detail what you are using the memory peek API for ? That might suggest a completely different libvirt API, or even something outside of libvirft
That is nothing special. I am working on something like virt-top, which requires to read VM's memory. I am also doing some experiments, which requires to get data out as fast as possible. Really I dont think that current multiple layers are the culprit of the slow problem. It must be the way we are reading memory out, as I wrote above. That is way to inefficient, and nobody can deny that. Thanks, J
On Tue, Jul 21, 2009 at 07:39:07PM +0900, Jun Koi wrote:
Hi,
I play around with MemoryPeek() API on QEMU. While it works well, I found that it is too slow.
That is expected because of the way it works: we always need to save memory to a file, and read it in again, and that is too inefficient.
Yes this is correct, and works that way because of how the qemu 'memsave' command works. Also there is a limit on the size of each peek (64K?) which is due to the libvirt remote protocol, specifically the max message size.
I am trying to figure out a better way to do this. To do that, clearly we need to re-architecture QEMU for this: We must have another way to read memory from outside the QEMU process.
Anybody could suggest a solution for this problem? I am willing to spend time on this feature to improve the situation.
A better virDomainMemoryPeek / QEMU memsize would have the following features: (1) Allow large chunks of memory to be snapshotted all in one go. We'd still have to fetch them piece-by-piece because of the max message size in the libvirt remote protocol, but at least the single large snapshot would be more consistent. (2) Don't use the temporary file (?) (3) Be faster, eg. in the non-remote case. (4) Work with Xen and other hypervisors ... (5) Work with physical memory, virtual memory and registers. Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On Wed, Jul 22, 2009 at 7:02 PM, Richard W.M. Jones<rjones@redhat.com> wrote:
On Tue, Jul 21, 2009 at 07:39:07PM +0900, Jun Koi wrote:
Hi,
I play around with MemoryPeek() API on QEMU. While it works well, I found that it is too slow.
That is expected because of the way it works: we always need to save memory to a file, and read it in again, and that is too inefficient.
Yes this is correct, and works that way because of how the qemu 'memsave' command works.
Also there is a limit on the size of each peek (64K?) which is due to the libvirt remote protocol, specifically the max message size.
I am trying to figure out a better way to do this. To do that, clearly we need to re-architecture QEMU for this: We must have another way to read memory from outside the QEMU process.
Anybody could suggest a solution for this problem? I am willing to spend time on this feature to improve the situation.
A better virDomainMemoryPeek / QEMU memsize would have the following features:
(1) Allow large chunks of memory to be snapshotted all in one go. We'd still have to fetch them piece-by-piece because of the max message size in the libvirt remote protocol, but at least the single large snapshot would be more consistent.
(2) Don't use the temporary file (?)
(3) Be faster, eg. in the non-remote case.
(4) Work with Xen and other hypervisors ...
(5) Work with physical memory, virtual memory and registers.
On (5): that requires another API, not this virDomainMemoryPeek() I agreed on (1), (2), (3) & (4). For now, I want to improve the situation on QEMU (thus, KVM) first. One obvious way is to transfer data directly from QEMU driver to QEMU using a particular IPC. How about having an Unix domain socket for this purpose? I imagine that we need to have a separate socket, opened and managed by QEMU interface. Idea? Thanks, J
On Thu, Jul 23, 2009 at 1:36 PM, Jun Koi<junkoi2004@gmail.com> wrote:
On Wed, Jul 22, 2009 at 7:02 PM, Richard W.M. Jones<rjones@redhat.com> wrote:
On Tue, Jul 21, 2009 at 07:39:07PM +0900, Jun Koi wrote:
Hi,
I play around with MemoryPeek() API on QEMU. While it works well, I found that it is too slow.
That is expected because of the way it works: we always need to save memory to a file, and read it in again, and that is too inefficient.
Yes this is correct, and works that way because of how the qemu 'memsave' command works.
Also there is a limit on the size of each peek (64K?) which is due to the libvirt remote protocol, specifically the max message size.
I am trying to figure out a better way to do this. To do that, clearly we need to re-architecture QEMU for this: We must have another way to read memory from outside the QEMU process.
Anybody could suggest a solution for this problem? I am willing to spend time on this feature to improve the situation.
A better virDomainMemoryPeek / QEMU memsize would have the following features:
(1) Allow large chunks of memory to be snapshotted all in one go. We'd still have to fetch them piece-by-piece because of the max message size in the libvirt remote protocol, but at least the single large snapshot would be more consistent.
(2) Don't use the temporary file (?)
(3) Be faster, eg. in the non-remote case.
(4) Work with Xen and other hypervisors ...
(5) Work with physical memory, virtual memory and registers.
On (5): that requires another API, not this virDomainMemoryPeek()
I mean the registers part need a new API. Thanks, J
On Thu, Jul 23, 2009 at 01:36:30PM +0900, Jun Koi wrote:
For now, I want to improve the situation on QEMU (thus, KVM) first. One obvious way is to transfer data directly from QEMU driver to QEMU using a particular IPC. How about having an Unix domain socket for this purpose?
I imagine that we need to have a separate socket, opened and managed by QEMU interface.
Possible, but it sounds tricky to implement from both ends. Remember that libvirt already offers the machinery for talking to the QEMU monitor, but not for making other connections to QEMU. When we do this for migration it has all sorts of additional complications. Note that with the current memsave interface in QEMU you can set up a named pipe in the filesystem, have QEMU write to that, and read from the other end, which is almost what you want. Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
participants (3)
-
Daniel P. Berrange -
Jun Koi -
Richard W.M. Jones