Hi,
In the past few weeks, I conducted performance tests to evaluate IOPS
(Input/Output Operations Per Second) performance for locally attached
disks. The original discussion started on the openstack-discuss mailing
list since all the tests were conducted within an OpenStack cloud
environment. However, I decided to initiate a discussion here, as it
appears that the performance differences might be rooted in lower-level
factors rather than the OpenStack platform itself. It seems that these
differences are closely related to variations in libvirt/qemu versions and
kernel configurations.
Ultimately, I discovered that the performance is significantly enhanced
when the hypervisor is deployed on top of Ubuntu 22.04LTS. Under this
setup, I was able to achieve around 100,000 IOPS during my fio tests
[1][2]. In contrast, conducting a similar test with the hypervisor deployed
on Ubuntu 20.04LTS yielded significantly lower results, averaging around
20,000 IOPS [3][4]. An intriguing observation is that attaching an nvme
disk to my Ubuntu 22.04LTS system and utilizing it as local storage led to
slightly diminished performance, hovering around 90,000 IOPS [5]. This
outcome is somewhat unexpected, as I initially anticipated higher figures.
It's particularly noteworthy that when I conduct the same test directly on
top of the hypervisor, the numbers align more closely with expectations
[6][7]. This pattern suggests that there might be a limitation imposed
either by libvirt/qemu or the kernel itself.
Having meticulously reviewed all the release notes, I failed to come across
any information pertaining to noteworthy performance enhancements
concerning local storage and IOPS. Given this, I'd like to reach out to you
directly to inquire if you possess any insights into such limitations. Any
guidance or suggestions that could help optimize my local storage results
would be greatly appreciated.
Looking forward to your input.
Best regards
/Jan Wasilewski
*References:[1] Configuration details(libvirt/qemu/kernel version) for
Ubuntu 22.04:
https://paste.openstack.org/show/b8svl0bOfX0WHTGvgI1h/
<
https://paste.openstack.org/show/b8svl0bOfX0WHTGvgI1h/>[2] fio results for
VM test with Ubuntu 22.04:
https://paste.openstack.org/show/bUpiq1y0o58S1ThNgqLd/
<
https://paste.openstack.org/show/bUpiq1y0o58S1ThNgqLd/>[3] Configuration
details(libvirt/qemu/kernel version) for Ubuntu 20.04:
https://paste.openstack.org/show/b005K2L6ZutGxDrCOZbL/
<
https://paste.openstack.org/show/b005K2L6ZutGxDrCOZbL/>[4] fio results for
VM test with Ubuntu 20.04:
https://paste.openstack.org/show/b8JeVOn4YCPSX7uaqR0N/
<
https://paste.openstack.org/show/b8JeVOn4YCPSX7uaqR0N/>[5] fio results for
VM test with Ubuntu 22.04 and nvme as a local storage disk:
https://paste.openstack.org/show/b75M4bI00LQePTmYSaUI/
<
https://paste.openstack.org/show/b75M4bI00LQePTmYSaUI/>[6] fio results for
hypervisor test with Ubuntu 22.04 and nvme storage:
https://paste.openstack.org/show/bj7NeSiwwiWS6MbJROts/
<
https://paste.openstack.org/show/bj7NeSiwwiWS6MbJROts/>[7] fio results for
hypervisor test with Ubuntu 22.04 and ssd storage:
https://paste.openstack.org/show/bSeEHXcbrY9YlYWQWGKS/
<
https://paste.openstack.org/show/bSeEHXcbrY9YlYWQWGKS/>*