[libvirt-users] data monitored by "virDomainBlockStats" API not consistent with ones reading from /proc/[pid]/io, and corrugated with /proc/diskstats
Hi, all. While monitoring KVM VMs’ disk utilisation using libvirt API “virDomainBlockStats", I find that the results acquired is not consistent with the value read from /proc/[vm_pid]/io(i only care about “rd_bytes" and “wr_bytes” fields). Since libvirt doesn’t support host level monitoring, i have to calculate the host’s disk utilisation by reading from the /proc/diskstats file. What’s worse, the individual VM’s disk data may be greater that the physical disk’s total value during some sampling time. It is supposed that the VM’s disk “rd_bytes" and “wr_bytes" should never exceed the value of the physical disk respectively, right ? Doesn’t libvirt virDomainBlockStats read disk information from “proc” ? I want to calculate VM’s bandwidth, here is my way: Firstly, we can get total disk utilisation using tools like iostat("iostat -x”, the %util field represent the disk’s utilisation percentage) Then, given a sampling time period, 10s for example, i collected total bytes read/written by each VM by virDomainBlockStats and total bytes of the physical disk by /proc/diskstats, then delta(VM’s rd+wr bytes)/delta(physical disk’s rd+wr bytes) * %util is the VM’s disk utilization percentage during this time period. Since the appearance of the above situation, i can’t get correct result. I have search a lot for this question, but no answers. Any help would be appreciated.
On 06/24/2014 02:24 AM, coperd wrote:
It is supposed that the VM’s disk “rd_bytes" and “wr_bytes" should
never exceed the value of the physical disk respectively, right ? That's a question for the qemu folks.
Doesn’t libvirt virDomainBlockStats read disk information from “proc” ?
No, libvirt reads it from the qemu QMP command query-blockstats. Beyond that, you'll have to investigate how qemu comes up with that number. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Just to clarify my problems: For each KVM virtual machine, it is actually a process running on the host. The $pid in "/proc/$pid/io" represents the VM process’s PID. All my work is done on the host, not inside the VMs. Take the VM process’s pid is “pid”, What the /proc/$pid/io records is io statistics of the VM from the host’s perspective. I understand that qemu might do some I/O coalescing kind of work, but /proc/$pid/io reveals the actual IOs submitted to the host kernel. For example, through “ps” command, i get the following info of one of my VMs: (the pid of the VM process is 4438) qemu 4438 1 0 Jun24 ? 00:06:25 /usr/bin/qemu-system-x86_64 ... [unrelevant data fields are ignored] Command line "virsh domblkstat $vmname vda” get the results below: vda rd_bytes 639415808 vda wr_bytes 728186880 Command line "cat /proc/4438/io” get the result below: read_bytes: 772415488 write_bytes: 734040064 I execute the above two commands consecutively using “&&", but we can that the results vary a lot, especially for the read_bytes. Since libvirt virDomainBlockStats API reads data from qemu i should ask the qemu folks for the answers. Great thanks, anyway. On Jun 25, 2014, at 3:45 AM, Eric Blake <eblake@redhat.com> wrote:
On 06/24/2014 02:24 AM, coperd wrote:
It is supposed that the VM’s disk “rd_bytes" and “wr_bytes" should
never exceed the value of the physical disk respectively, right ?
That's a question for the qemu folks.
Doesn’t libvirt virDomainBlockStats read disk information from “proc” ?
No, libvirt reads it from the qemu QMP command query-blockstats. Beyond that, you'll have to investigate how qemu comes up with that number.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 06/25/2014 05:11 AM, coperd wrote: [please don't top-post on technical lists]
Command line "virsh domblkstat $vmname vda” get the results below: vda rd_bytes 639415808 vda wr_bytes 728186880
Command line "cat /proc/4438/io” get the result below: read_bytes: 772415488 write_bytes: 734040064
Bear in mind that for some file formats, such as qcow2, qemu must read (and write) additional I/O to manage the metadata of the file in comparison to the amount of read and writes performed on behalf of the guest; the qemu numbers are reported from the guest perspective, not what qemu asked the host kernel to do. So seeing /proc/NN/io have larger numbers than qemu makes total sense. I'm still not clear how you are getting vcpu PID numbers for reading from /proc; it's not something that libvirt is currently advertising. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Jun 26, 2014, at 2:50 AM, Eric Blake <eblake@redhat.com> wrote:
I'm still not clear how you are getting vcpu PID numbers for reading from /proc; it's not something that libvirt is currently advertising.
I thought the VM process’s /proc statistics information is exactly the same as the guest’s behaviour. It turns out to be my misunderstanding. Each vcpu is a thread of the VM process in qemu implementation, whose thread id can be obtained by the “-L” option of the “ps” command. It seems that I don't mention the vcpu PID. I’m referring to the VM process’s /proc/$pid. If running a VM whose image file is located on the local disk, how can i calculate the disk bandwidth(the percentage, e.g. 40% disk) used by the VM?
participants (2)
-
coperd -
Eric Blake