[libvirt-users] Yet another disk I/O performance issue
Hi all, I'm running a VM using libvirt+KVM and I have a disk performance issue. The host is the following: 4 cores Intel Xeon 5140@2.33 GHz, 16 GB of RAM, SATA HDD, OS Debian Wheezy, libvirt 0.9.12-11, QEMU-KVM 1.1.2+dfsg-2. The guest: 1 CPU, 2 GB RAM running Debian 7.0, image in compressed qcow2 format. When I try do run "dd if=/dev/zero of=io.test bs=32768k count=40" I get around 500 MB/s on bare metal, while only around 30 MB/s inside the VM. I'm trying to get something more out of the virtualization layer, I hope that there's room for improvement. I'm using virtio, I aready set cache='none' and io='native' in the domain definition. Both host and guest are using deadline as I/O scheduler. The VM uses an ext4 filesystem, while the image is saved on an ext3 disk. I mounted the host and guest filesystems specifying nodiratime and noatime options. Even if I convert the image to raw format nothing changes. I didn't mess with iotune nor blockio. Is there something that I overlooked or any other suggestion? Thanks in advance for your help. Matteo -- A refund for defective software might be nice, except it would bankrupt the entire software industry in the first year. Andrew S. Tanenbaum, Computer Networks, 2003, Introduction, page 14 Linux registered user #463400
On Sun, Jan 19, 2014 at 08:01:55PM +0100, Matteo Lanati wrote:
Hi all,
I'm running a VM using libvirt+KVM and I have a disk performance issue.
The host is the following: 4 cores Intel Xeon 5140@2.33 GHz, 16 GB of RAM, SATA HDD, OS Debian Wheezy, libvirt 0.9.12-11, QEMU-KVM 1.1.2+dfsg-2.
The guest: 1 CPU, 2 GB RAM running Debian 7.0, image in compressed qcow2 format.
When I try do run "dd if=/dev/zero of=io.test bs=32768k count=40" I get around 500 MB/s on bare metal, while only around 30 MB/s inside the VM.
'dd' is rarely a meaningful performance benchmark. In particular with the way you're invoking it, you're not requesting that the OS flush data to disk - so the data will be building up in RAM which is very fast. Since your host likely has much more free RAM for I/O cache than the guest, dd in the host is able todo much more I/O to fast RAM before the OS has to start actually writing to slow disk. The guest which is likely writing to slow disk much sooner giving the lower results. At the very least you need to request use of O_DIRECT with dd so it is writing straight to disk. Or try a formal benchmark tool like iozone Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hi Daniel, thank you very much for your explanation. Now every fits into the picture. I run the tests you suggested and the numbers make much more sense. Thanks for stating the obvious but it was the first time I needed to compare I/O. Best regards, Matteo On 20 Jan 2014, at 15:19, Daniel P. Berrange <berrange@redhat.com> wrote:
'dd' is rarely a meaningful performance benchmark.
In particular with the way you're invoking it, you're not requesting that the OS flush data to disk - so the data will be building up in RAM which is very fast. Since your host likely has much more free RAM for I/O cache than the guest, dd in the host is able todo much more I/O to fast RAM before the OS has to start actually writing to slow disk. The guest which is likely writing to slow disk much sooner giving the lower results.
At the very least you need to request use of O_DIRECT with dd so it is writing straight to disk. Or try a formal benchmark tool like iozone
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
A refund for defective software might be nice, except it would bankrupt the entire software industry in the first year. Andrew S. Tanenbaum, Computer Networks, 2003, Introduction, page 14 Linux registered user #463400
Hey, First of all, you don't need to set scheduler as deadline on VM. Just disable it (elevator=none). The HV will handle it for the VM. Furthermore, as generic performance optimization, I recommend you disable swappiness on the HV, as well as zone reclaim. I invite you to read this IBM paper: http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaat/liaa... It's quite a good starting point. Regards, On 01/19/2014 08:01 PM, Matteo Lanati wrote:
Hi all,
I'm running a VM using libvirt+KVM and I have a disk performance issue.
The host is the following: 4 cores Intel Xeon 5140@2.33 GHz, 16 GB of RAM, SATA HDD, OS Debian Wheezy, libvirt 0.9.12-11, QEMU-KVM 1.1.2+dfsg-2.
The guest: 1 CPU, 2 GB RAM running Debian 7.0, image in compressed qcow2 format.
When I try do run "dd if=/dev/zero of=io.test bs=32768k count=40" I get around 500 MB/s on bare metal, while only around 30 MB/s inside the VM.
I'm trying to get something more out of the virtualization layer, I hope that there's room for improvement.
I'm using virtio, I aready set cache='none' and io='native' in the domain definition. Both host and guest are using deadline as I/O scheduler. The VM uses an ext4 filesystem, while the image is saved on an ext3 disk. I mounted the host and guest filesystems specifying nodiratime and noatime options. Even if I convert the image to raw format nothing changes. I didn't mess with iotune nor blockio.
Is there something that I overlooked or any other suggestion?
Thanks in advance for your help.
Matteo
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
-- Pierre Schweitzer <pierre@reactos.org> System Administrator ReactOS Foundation
participants (3)
-
Daniel P. Berrange -
Matteo Lanati -
Pierre Schweitzer