[libvirt-users] memoryStats question
Hello, I'm trying to get memory usage statistic inside a guest machine using virDomainMemoryStats function at libvirt. According to http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryStatTags the structure could have a lot of useful memory statistic but the amount of parameters depends on the hypervisor and the driver. However using KVM I'm able to get VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON and VIR_DOMAIN_MEMORY_STAT_RSS only. Am I right it is not possible to get additional info such as used/unused memory inside a guest machine in case of KVM?
On Thu, Jul 03, 2014 at 04:31:46PM +0300, Gleb Voronich wrote:
Hello,
I'm trying to get memory usage statistic inside a guest machine using virDomainMemoryStats function at libvirt. According to http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryStatTags the structure could have a lot of useful memory statistic but the amount of parameters depends on the hypervisor and the driver. However using KVM I'm able to get VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON and VIR_DOMAIN_MEMORY_STAT_RSS only.
QEMU/KVM only supports these two IIRC.
Am I right it is not possible to get additional info such as used/unused memory inside a guest machine in case of KVM?
Yes. Actually reporting some data from *inside* the domain is not what libvirt aims to provide. Martin
On 07/03/2014 07:58 AM, Martin Kletzander wrote:
On Thu, Jul 03, 2014 at 04:31:46PM +0300, Gleb Voronich wrote:
Hello,
I'm trying to get memory usage statistic inside a guest machine using virDomainMemoryStats function at libvirt. According to http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryStatTags the structure could have a lot of useful memory statistic but the amount of parameters depends on the hypervisor and the driver. However using KVM I'm able to get VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON and VIR_DOMAIN_MEMORY_STAT_RSS only.
QEMU/KVM only supports these two IIRC.
Am I right it is not possible to get additional info such as used/unused memory inside a guest machine in case of KVM?
Yes. Actually reporting some data from *inside* the domain is not what libvirt aims to provide.
In the case of the memory balloon device, libvirt CAN provide stats from the guest, because of the way the balloon device works. But if you aren't using a balloon, or if the guest is not cooperative, this won't work. Better would be a guest-agent command for getting at this information (but again, you are relying on guest cooperation). In short, libvirt is great at stats from the HOST point of view, but for stats from the GUEST point of view, you are often better off using the same mechanisms that you would for getting stats from a remote bare metal machine, without needing libvirt in the mix. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Eric Blake <mailto:eblake@redhat.com>
In the case of the memory balloon device, libvirt CAN provide stats from the guest, because of the way the balloon device works. But if you aren't using a balloon, or if the guest is not cooperative, this won't work. Better would be a guest-agent command for getting at this information (but again, you are relying on guest cooperation). In short, libvirt is great at stats from the HOST point of view, but for stats from the GUEST point of view, you are often better off using the same mechanisms that you would for getting stats from a remote bare metal machine, without needing libvirt in the mix.
Thanks Eric. 1. I use the balloon for all of my guests: <memballoon model='virtio'> But how can I get stats? 2. I've tried the latest QEMU GA 2.0.0
On 07/03/2014 08:29 AM, Gleb Voronich wrote:
Eric Blake <mailto:eblake@redhat.com>
In the case of the memory balloon device, libvirt CAN provide stats from the guest, because of the way the balloon device works. But if you aren't using a balloon, or if the guest is not cooperative, this won't work. Better would be a guest-agent command for getting at this information (but again, you are relying on guest cooperation). In short, libvirt is great at stats from the HOST point of view, but for stats from the GUEST point of view, you are often better off using the same mechanisms that you would for getting stats from a remote bare metal machine, without needing libvirt in the mix.
Thanks Eric.
1. I use the balloon for all of my guests: <memballoon model='virtio'> But how can I get stats?
virDomainMemoryStats() gets those stats, if qemu is new enough to provide them, and if the guest cooperates to provide them.
2. I've tried the latest QEMU GA 2.0.0
Right now, there are no memory stats available through the guest agent, only through the balloon device. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
virDomainMemoryStats() gets those stats, if qemu is new enough to provide them, and if the guest cooperates to provide them. Well I use the latest QEMU 2.0.0 I have the latest CentOS 6.5 installed on the guest and unfortunately I can't get more stats that I have.
Which OS are you using in your guests?
Of course, you need to have the virtio balloon driver up and runnig in your guests. CentOS 6.5 x86_64 It has the balloon driver and it is up and running:
# modinfo virtio_balloon filename: /lib/modules/2.6.32-431.17.1.el6.x86_64/kernel/drivers/virtio/virtio_balloon.ko license: GPL description: Virtio balloon driver srcversion: BB9F75B2CAF3435CC507998 alias: virtio:d00000005v* depends: virtio,virtio_ring vermagic: 2.6.32-431.17.1.el6.x86_64 SMP mod_unload modversions
On 07/03/2014 08:46 AM, Gleb Voronich wrote:
virDomainMemoryStats() gets those stats, if qemu is new enough to provide them, and if the guest cooperates to provide them. Well I use the latest QEMU 2.0.0 I have the latest CentOS 6.5 installed on the guest and unfortunately I can't get more stats that I have.
Then your problem is your libvirt is too old. It is the combination of qemu new enough to provide stats (qemu 1.4; probably not present in the qemu shipped in CentOS, but present in your self-built 2.0) and libvirt new enough to read stats (v1.1.1 or newer; which is in RHEL 7.0, but not backported to the libvirt 0.10.2 of RHEL 6.5, and therefore not in your CentOS setup). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Then your problem is your libvirt is too old. It is the combination of qemu new enough to provide stats (qemu1.4; probably not present in the qemu shipped in CentOS, but present in your self-built 2.0) and libvirt new enough to read stats (v1.1.1 or newer; which is in RHEL 7.0, but not backported to the libvirt0.10.2 of RHEL6.5, and therefore not in your CentOS setup).
Eric could you please let me know how I can debug the exact reason of this issue? I've build qemu 2.0 and libvirt 1.2.5 for CentOS 6.5 x86_64 and I use 3.10.44 kernel from elrepo (kernel-lt). According to your words this environment should provide more memory statistic. Is there a way I can debug the call using gdb or some other tool? I do really need to get this working. Thanks.
At Thu, 03 Jul 2014 17:46:14 +0300, Gleb Voronich wrote:
virDomainMemoryStats() gets those stats, if qemu is new enough to provide them, and if the guest cooperates to provide them. Well I use the latest QEMU 2.0.0 I have the latest CentOS 6.5 installed on the guest and unfortunately I can't get more stats that I have.
Which OS are you using in your guests?
Of course, you need to have the virtio balloon driver up and runnig in your guests. CentOS 6.5 x86_64 It has the balloon driver and it is up and running:
# modinfo virtio_balloon filename: /lib/modules/2.6.32-431.17.1.el6.x86_64/kernel/drivers/virtio/virtio_balloon.ko license: GPL description: Virtio balloon driver srcversion: BB9F75B2CAF3435CC507998 alias: virtio:d00000005v* depends: virtio,virtio_ring vermagic: 2.6.32-431.17.1.el6.x86_64 SMP mod_unload modversions
In this case, it's your guest kernel that's too old to support these stats. Apparently, reporting statistics in the balloon driver was introduced in the kernel on 24th Feb 2010 (commit 9564e138b1f6eb137f7149772438d3f3fb3277dd) which clearly predates version 2.6.32 (2nd December 2009). Claudio
On 07/04/2014 03:18 PM, Claudio Bley wrote:
At Thu, 03 Jul 2014 17:46:14 +0300, Gleb Voronich wrote:
virDomainMemoryStats() gets those stats, if qemu is new enough to provide them, and if the guest cooperates to provide them. Well I use the latest QEMU 2.0.0 I have the latest CentOS 6.5 installed on the guest and unfortunately I can't get more stats that I have.
Which OS are you using in your guests?
Of course, you need to have the virtio balloon driver up and runnig in your guests. CentOS 6.5 x86_64 It has the balloon driver and it is up and running:
# modinfo virtio_balloon filename: /lib/modules/2.6.32-431.17.1.el6.x86_64/kernel/drivers/virtio/virtio_balloon.ko license: GPL description: Virtio balloon driver srcversion: BB9F75B2CAF3435CC507998 alias: virtio:d00000005v* depends: virtio,virtio_ring vermagic: 2.6.32-431.17.1.el6.x86_64 SMP mod_unload modversions
In this case, it's your guest kernel that's too old to support these stats.
Apparently, reporting statistics in the balloon driver was introduced in the kernel on 24th Feb 2010 (commit 9564e138b1f6eb137f7149772438d3f3fb3277dd) which clearly predates version 2.6.32 (2nd December 2009).
Acording to the kernel RPM's changelog, this was backported into 2.6.32-38.el6 and released in RHEL 6.0 (Nov 2010): https://bugzilla.redhat.com/show_bug.cgi?id=601690 Jan
At Fri, 04 Jul 2014 16:18:45 +0200, Ján Tomko wrote:
[1 <text/plain; ISO-8859-1 (quoted-printable)>] On 07/04/2014 03:18 PM, Claudio Bley wrote:
At Thu, 03 Jul 2014 17:46:14 +0300, Gleb Voronich wrote:
virDomainMemoryStats() gets those stats, if qemu is new enough to provide them, and if the guest cooperates to provide them. Well I use the latest QEMU 2.0.0 I have the latest CentOS 6.5 installed on the guest and unfortunately I can't get more stats that I have.
Which OS are you using in your guests?
Of course, you need to have the virtio balloon driver up and runnig in your guests. CentOS 6.5 x86_64 It has the balloon driver and it is up and running:
# modinfo virtio_balloon filename: /lib/modules/2.6.32-431.17.1.el6.x86_64/kernel/drivers/virtio/virtio_balloon.ko license: GPL description: Virtio balloon driver srcversion: BB9F75B2CAF3435CC507998 alias: virtio:d00000005v* depends: virtio,virtio_ring vermagic: 2.6.32-431.17.1.el6.x86_64 SMP mod_unload modversions
In this case, it's your guest kernel that's too old to support these stats.
Apparently, reporting statistics in the balloon driver was introduced in the kernel on 24th Feb 2010 (commit 9564e138b1f6eb137f7149772438d3f3fb3277dd) which clearly predates version 2.6.32 (2nd December 2009).
Acording to the kernel RPM's changelog, this was backported into 2.6.32-38.el6 and released in RHEL 6.0 (Nov 2010): https://bugzilla.redhat.com/show_bug.cgi?id=601690
In that case, I'd guess that Gleb just forgot to call virDomainSetMemoryStatsPeriod, perhaps? Gleb, you didn't mention what language (binding) you're using. In virsh, use the "--period N" option for the dommemstat command at least once in order to enable stats collection. -- Claudio
In that case, I'd guess that Gleb just forgot to call virDomainSetMemoryStatsPeriod, perhaps? I've just performed the same test with CentOS 7 at the host and CentOS 7 at the guest. Everything works.
# virsh dommemstat vm1 actual 1048576 swap_in 0 swap_out 0 major_fault 751 minor_fault 3958627 unused 375900 available 1017988 rss 978248 I don't know the reason it doesn't work with ELRepo 3.10.45 kernel but I'd really like to. According to https://bugzilla.redhat.com/show_bug.cgi?id=601690 memoryStats should work at host's side starting from 2.6.32-52.el6 so I suppose the problem is the guest kernel. Do you think this could be caused by an old version of virtio_balloon driver?
At Tue, 08 Jul 2014 18:21:54 +0300, Gleb Voronich wrote:
In that case, I'd guess that Gleb just forgot to call virDomainSetMemoryStatsPeriod, perhaps? I've just performed the same test with CentOS 7 at the host and CentOS 7 at the guest. Everything works.
# virsh dommemstat vm1 actual 1048576 swap_in 0 swap_out 0 major_fault 751 minor_fault 3958627 unused 375900 available 1017988 rss 978248
I don't know the reason it doesn't work with ELRepo 3.10.45 kernel but I'd really like to. According to https://bugzilla.redhat.com/show_bug.cgi?id=601690 memoryStats should work at host's side starting from 2.6.32-52.el6 so I suppose the problem is the guest kernel.
Do you think this could be caused by an old version of virtio_balloon driver?
Maybe. It's hard to tell without exactly knowing what you did... Try turning on debugging output for libvirtd in order dig into this. But, since it works in general, it rules out libvirt to be at fault here. You may have better luck asking on the qemu-user list. For me, it works with kernel 2.6.32-431.el6.x86_64 from CentOS 6.5. -- Claudio
At Thu, 3 Jul 2014 15:58:41 +0200, Martin Kletzander wrote:
On Thu, Jul 03, 2014 at 04:31:46PM +0300, Gleb Voronich wrote:
Hello,
I'm trying to get memory usage statistic inside a guest machine using virDomainMemoryStats function at libvirt. According to http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryStatTags the structure could have a lot of useful memory statistic but the amount of parameters depends on the hypervisor and the driver. However using KVM I'm able to get VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON and VIR_DOMAIN_MEMORY_STAT_RSS only.
QEMU/KVM only supports these two IIRC.
Actually, you should get more stats when using the virtio balloon driver in your guests. E.g. the windows virtio driver supports these stats: SWAP_IN 0 /* Amount of memory swapped in */ SWAP_OUT 1 /* Amount of memory swapped out */ MAJFLT 2 /* Number of major faults */ MINFLT 3 /* Number of minor faults */ MEMFREE 4 /* Total amount of free memory */ MEMTOT 5 /* Total amount of memory */ See https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/9a5cb6749d68... IFAIR, these should be queried by the virDomainMemoryStats function. Claudio -- BSc (Comp) Claudio Bley - Principal Software Engineer AV-TEST GmbH, Klewitzstr. 7, 39112 Magdeburg, Germany Phone: +49 391 6075460, Fax: +49 391 6075469 Web: <http://www.av-test.org> * https://twitter.com/avtestorg * https://facebook.com/avtestorg * * https://plus.google.com/100383867141221115206/ * Eingetragen am / Registered at: Amtsgericht Stendal (HRB 114076) Geschaeftsfuehrer (CEO): Andreas Marx, Guido Habicht, Maik Morgenstern Our services shall be effected on the basis of the General Terms and Conditions of AV-TEST GmbH, which are accessible under <http://www.av-test.org/en/av-test/terms-and-conditions/> or obtainable upon request. Unsere Leistungen erfolgen auf der Grundlage der Allgemeinen Geschäftsbedingungen der AV-TEST GmbH, die unter <http://www.av-test.org/av-test/agb/> abrufbar sind oder auf Anfrage übersandt werden.
At Thu, 03 Jul 2014 17:32:20 +0300, Gleb Voronich wrote:
Actually, you should get more stats when using the virtio balloon driver in your guests. I use the balloon but I can't get more stats. I use python binding for libvirt and as far as I can see it calls virDomainMemoryStats exactly.
Which OS are you using in your guests? Of course, you need to have the virtio balloon driver up and runnig in your guests. Claudio
On 07/03/2014 08:39 AM, Claudio Bley wrote:
At Thu, 03 Jul 2014 17:32:20 +0300, Gleb Voronich wrote:
Actually, you should get more stats when using the virtio balloon driver in your guests. I use the balloon but I can't get more stats. I use python binding for libvirt and as far as I can see it calls virDomainMemoryStats exactly.
Which OS are you using in your guests?
Of course, you need to have the virtio balloon driver up and runnig in your guests.
You also have to have new enough qemu that reports stats without blocking (qemu 0.14 or newer added stat reporting via qom-get, since the older query-balloon interface blocks) and libvirt that knows how to read stats via qom-get (libvirt 1.1.1 or newer). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
You also have to have new enough qemu that reports stats without blocking (qemu 0.14 or newer added stat reporting via qom-get, since the older query-balloon interface blocks) and libvirt that knows how to read stats via qom-get (libvirt 1.1.1 or newer)
As I've already mentioned I use QEMU 2.0.0 and Libvirt 1.2.5 with no luck :(
participants (6)
-
Claudio Bley -
Eric Blake -
Gleb Voronich -
Ján Tomko -
Martin Kletzander -
Ok