Performance Discrepancies and Limitations in Local Storage IOPS Testing

Hi, In the past few weeks, I conducted performance tests to evaluate IOPS (Input/Output Operations Per Second) performance for locally attached disks. The original discussion started on the openstack-discuss mailing list since all the tests were conducted within an OpenStack cloud environment. However, I decided to initiate a discussion here, as it appears that the performance differences might be rooted in lower-level factors rather than the OpenStack platform itself. It seems that these differences are closely related to variations in libvirt/qemu versions and kernel configurations. Ultimately, I discovered that the performance is significantly enhanced when the hypervisor is deployed on top of Ubuntu 22.04LTS. Under this setup, I was able to achieve around 100,000 IOPS during my fio tests [1][2]. In contrast, conducting a similar test with the hypervisor deployed on Ubuntu 20.04LTS yielded significantly lower results, averaging around 20,000 IOPS [3][4]. An intriguing observation is that attaching an nvme disk to my Ubuntu 22.04LTS system and utilizing it as local storage led to slightly diminished performance, hovering around 90,000 IOPS [5]. This outcome is somewhat unexpected, as I initially anticipated higher figures. It's particularly noteworthy that when I conduct the same test directly on top of the hypervisor, the numbers align more closely with expectations [6][7]. This pattern suggests that there might be a limitation imposed either by libvirt/qemu or the kernel itself. Having meticulously reviewed all the release notes, I failed to come across any information pertaining to noteworthy performance enhancements concerning local storage and IOPS. Given this, I'd like to reach out to you directly to inquire if you possess any insights into such limitations. Any guidance or suggestions that could help optimize my local storage results would be greatly appreciated. Looking forward to your input. Best regards /Jan Wasilewski *References:[1] Configuration details(libvirt/qemu/kernel version) for Ubuntu 22.04: https://paste.openstack.org/show/b8svl0bOfX0WHTGvgI1h/ <https://paste.openstack.org/show/b8svl0bOfX0WHTGvgI1h/>[2] fio results for VM test with Ubuntu 22.04: https://paste.openstack.org/show/bUpiq1y0o58S1ThNgqLd/ <https://paste.openstack.org/show/bUpiq1y0o58S1ThNgqLd/>[3] Configuration details(libvirt/qemu/kernel version) for Ubuntu 20.04: https://paste.openstack.org/show/b005K2L6ZutGxDrCOZbL/ <https://paste.openstack.org/show/b005K2L6ZutGxDrCOZbL/>[4] fio results for VM test with Ubuntu 20.04: https://paste.openstack.org/show/b8JeVOn4YCPSX7uaqR0N/ <https://paste.openstack.org/show/b8JeVOn4YCPSX7uaqR0N/>[5] fio results for VM test with Ubuntu 22.04 and nvme as a local storage disk: https://paste.openstack.org/show/b75M4bI00LQePTmYSaUI/ <https://paste.openstack.org/show/b75M4bI00LQePTmYSaUI/>[6] fio results for hypervisor test with Ubuntu 22.04 and nvme storage: https://paste.openstack.org/show/bj7NeSiwwiWS6MbJROts/ <https://paste.openstack.org/show/bj7NeSiwwiWS6MbJROts/>[7] fio results for hypervisor test with Ubuntu 22.04 and ssd storage: https://paste.openstack.org/show/bSeEHXcbrY9YlYWQWGKS/ <https://paste.openstack.org/show/bSeEHXcbrY9YlYWQWGKS/>*