- Users - libvirt List Archives

Performance Discrepancies and Limitations in Local Storage IOPS Testing
by Jan Wasilewski 24 Aug '23

24 Aug '23

Hi, In the past few weeks, I conducted performance tests to evaluate IOPS (Input/Output Operations Per Second) performance for locally attached disks. The original discussion started on the openstack-discuss mailing list since all the tests were conducted within an OpenStack cloud environment. However, I decided to initiate a discussion here, as it appears that the performance differences might be rooted in lower-level factors rather than the OpenStack platform itself. It seems that these differences are closely related to variations in libvirt/qemu versions and kernel configurations. Ultimately, I discovered that the performance is significantly enhanced when the hypervisor is deployed on top of Ubuntu 22.04LTS. Under this setup, I was able to achieve around 100,000 IOPS during my fio tests [1][2]. In contrast, conducting a similar test with the hypervisor deployed on Ubuntu 20.04LTS yielded significantly lower results, averaging around 20,000 IOPS [3][4]. An intriguing observation is that attaching an nvme disk to my Ubuntu 22.04LTS system and utilizing it as local storage led to slightly diminished performance, hovering around 90,000 IOPS [5]. This outcome is somewhat unexpected, as I initially anticipated higher figures. It's particularly noteworthy that when I conduct the same test directly on top of the hypervisor, the numbers align more closely with expectations [6][7]. This pattern suggests that there might be a limitation imposed either by libvirt/qemu or the kernel itself. Having meticulously reviewed all the release notes, I failed to come across any information pertaining to noteworthy performance enhancements concerning local storage and IOPS. Given this, I'd like to reach out to you directly to inquire if you possess any insights into such limitations. Any guidance or suggestions that could help optimize my local storage results would be greatly appreciated. Looking forward to your input. Best regards /Jan Wasilewski *References:[1] Configuration details(libvirt/qemu/kernel version) for Ubuntu 22.04: https://paste.openstack.org/show/b8svl0bOfX0WHTGvgI1h/ <https://paste.openstack.org/show/b8svl0bOfX0WHTGvgI1h/>[2] fio results for VM test with Ubuntu 22.04: https://paste.openstack.org/show/bUpiq1y0o58S1ThNgqLd/ <https://paste.openstack.org/show/bUpiq1y0o58S1ThNgqLd/>[3] Configuration details(libvirt/qemu/kernel version) for Ubuntu 20.04: https://paste.openstack.org/show/b005K2L6ZutGxDrCOZbL/ <https://paste.openstack.org/show/b005K2L6ZutGxDrCOZbL/>[4] fio results for VM test with Ubuntu 20.04: https://paste.openstack.org/show/b8JeVOn4YCPSX7uaqR0N/ <https://paste.openstack.org/show/b8JeVOn4YCPSX7uaqR0N/>[5] fio results for VM test with Ubuntu 22.04 and nvme as a local storage disk: https://paste.openstack.org/show/b75M4bI00LQePTmYSaUI/ <https://paste.openstack.org/show/b75M4bI00LQePTmYSaUI/>[6] fio results for hypervisor test with Ubuntu 22.04 and nvme storage: https://paste.openstack.org/show/bj7NeSiwwiWS6MbJROts/ <https://paste.openstack.org/show/bj7NeSiwwiWS6MbJROts/>[7] fio results for hypervisor test with Ubuntu 22.04 and ssd storage: https://paste.openstack.org/show/bSeEHXcbrY9YlYWQWGKS/ <https://paste.openstack.org/show/bSeEHXcbrY9YlYWQWGKS/>*

1 0

Warning : Failed to set up UEFI / The Libvirt version does not support UEFI / Install options are limited...
by Mario Marietto 22 Aug '23

22 Aug '23

Hello to everyone. I'm trying to use qemu 5.1 with virt-manager and libvirt on my ARM chromebook (armhf 32 bit cpu) running with Devuan 4 as host o.s. By default it uses qemu and its dependencies,version 5.2. I remember that I can't use qemu 5.2,because it doesn't have any support for KVM as you can read here : https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg02074.html For this reason,I've compiled qemu 5.1 from source. Below I shown how I have configured everything such as a little piece of compilation messages : # apt install libgtk-3-dev libpulse-dev libgbm-dev libspice-protocol-dev libspice-server-dev libusb-1.0-0-dev libepoxy-dev # cd /usr/share # mv qemu qemu_ # cd /usr/lib # mv qemu qemu_ # cd /usr/lib/arm-linux-gnueabihf # mv qemu qemu_ # cd /usr/lib/ipxe # mv qemu qemu_ # cd /usr/share/bash-completion/completions/ # mv qemu qemu_ # mv qemu-kvm qemu-kvm_ # mv qemu-system-i386 qemu-system-i386_ # mv qemu-system-x86_64 qemu-system-x86_64_ # cd /usr/bin # mv qemu-system-arm qemu-system-arm_ # cp /root/Desktop/qemu-v5.1.0/arm-softmmu/qemu-system-arm /usr/bin # CFLAGS=-Wno-error ./configure --target-list=x86_64-softmmu --enable-opengl --enable-gtk --enable-kvm --enable-guest-agent --enable-spice --audio-drv- list="oss pa" --enable-libusb A little piece of the log messages that I've got from the compilation of qemu 5.1 : https://pastebin.ubuntu.com/p/8DYfgPvhXy/ These are the resulting versions of my frankenstein operation : # virsh version Compiled against library: libvirt 7.0.0 Using library: libvirt 7.0.0 Using API: QEMU 7.0.0 Running hypervisor: QEMU 5.1.0 At this point I've run virt-manager. It has been able to detect qemu,but I get the following error : Warning : Failed to set up UEFI. The Libvirt version does not support UEFI. Install options are limited. Do you have some suggestions to give me to fix this error ? I'm sure to have missed something, thanks. -- Mario.

1 0

Does libvirt support intra-host KVM migration?
by Eric Wheeler 18 Aug '23

18 Aug '23

Hello all, I'm reposting this to the libvirt-users list: I looked around for documentation on intra-host KVM migration but haven't found much. For example, this could be useful to "migrate" VM to run on an upgraded version of `qemu-kvm` without migrating to a different host and migrating back. We tested migrating a VM to the same host on an old version of libvirt (el7), and it complained about UUID conflicts if the destination already hosts the same VM. Changing the name or the UUID in the destination XML during migration (`virsh migrate --xml`) gives an error that the UUID or name does not match during migration: UUID change: error: unsupported configuration: Target domain uuid 66a113f4-a101-4db1-8478-49cf088fedb9 does not matchsource b5256b25-e137-45b8-bddb-78545ab55fc4 Name change: error: unsupported configuration: Target domain name 'diskless2' does not match source 'diskless' Questions before we do more testing: - Do modern versions of libvirt support intra-host migration? - If so, which version? - Documentation? Any help you can provide would be greatly appreciated! -- Eric Wheeler

2 1

You will need to grant the 'libvirt-qemu' user search permissions for the following directories....
by Sebastien WILLEMIJNS 09 Aug '23

09 Aug '23

Hello, Why LIBVIRT software/libs need to chown "near the root level" (home/blahblah/) when raw/vdi/vhd can contains lots of directories as /home/user/Virtual_HDs/desktop/daddy/private/bedroom/number2/hd.vdi ? on ubuntu, "/media/hostname" can contains all our external HD's without relation with virtualization !!! :-( another sample picked up in the net: WARNING /home/jwright/virtualMachines/images/fedora25.qcow2 may not be accessible by the hypervisor. You will need to grant the 'qemu' user search permissions for the following directories: ['/home/jwright']

2 2

ipv6 can not work for direct type interface
by Yalan Zhang 07 Aug '23

07 Aug '23

Hi there, I have a question regarding direct type interfaces. Would someone be able to take a look at it? When I start 2 VMs on the same host with interface "direct type + bridge mode", just as below: <interface type="direct"> <mac address="52:54:00:9e:7b:51"/> <source dev="eno1" mode="bridge"/> <model type="virtio"/> </interface> The 2 VMs can connect to each other via ipv4, but can not connect to each other via ipv6. Maybe it's related to some kernel parameters, but I don't know how to debug. Is there anyone who can help me? Thank you! BR, Yalan

1 1

Security Logs
by Simon Fairweather 23 Jul '23

23 Jul '23

Hi Is there a way to enable logging of security revents, i.e. a remote client connecting to VNC/SPICE for console. I cannot see/find any documentation, I have also requested a feature request on QEMU gitlab. Regards Simon.

1 0

Issue with libvirtd service and virsh on Debian 13 "trixie" (testing)
by pascal 15 Jul '23

15 Jul '23

Hi, I just upgraded to Debian 13 and observing an issue related to libvirtd service. It happens during up time, the service is initially running and all is working fine. I can run virt-manager and use VMs. Then close virt-manager. After some time (5min), here is what happens : - virt-manager keeps on "Connecting... " when launching it. - "sudo virsh net-list --all" command hangs and does not respond. I have to type Ctrl-C. At that step, I need to restart the libvirtd service : "sudo systemctl restart libvirtd" Then everything is back and running as expected... For 5 minutes as issue comes back. All logs here : https://pastebin.com/qHQWLF4i -=- Versions used : sudo apt policy qemu-system Installé : 1:8.0.2+dfsg-3 sudo apt policy libvirt-daemon-system Installé : 9.5.0-1 $ sudo apt policy virt-manager Installé : 1:4.1.0-2 Can you help here ? Thanks and regards, Pascal

2 1

Cannot access guest with bridged networking when using firewalld (nftables backend) on Debian 12 Bookworm
by Niccolò Belli 12 Jul '23

12 Jul '23

firewalld version 1.3.0-1 libvirt version 9.0.0-4 network-manager version 1.42.4-1 # firewall-cmd --get-active-zones libvirt interfaces: br28 public interfaces: dac0 dac0.100 dac0.28 ftth # firewall-cmd --list-all --zone=public public (active) target: default icmp-block-inversion: no interfaces: dac0 dac0.100 dac0.28 ftth sources: services: dhcpv6-client ssh ports: protocols: forward: yes masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: dac0 is a Direct Attach cable with several vlans. dac0.100 is the vlan where I create the ppoe connection from my FTTH provider. ftth is the ppp name. dac0.28 is the vlan for the public /28 IPv4 subnet. br28 is the bridge where dac0.28 is attached. # brctl show bridge name bridge id STP enabled interfaces br28 8000.d2605c025b1d no dac0.28 vnet1 # firewall-cmd --list-all --zone=libvirt libvirt (active) target: ACCEPT icmp-block-inversion: no interfaces: br28 sources: services: dhcp dhcpv6 dns ssh tftp ports: protocols: icmp ipv6-icmp forward: no masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: rule priority="32767" reject # nft list tables table inet firewalld table ip mangle # nft list table ip mangle # Warning: table ip mangle is managed by iptables-nft, do not touch! table ip mangle { chain FORWARD { type filter hook forward priority mangle; policy accept; oifname "ftth" tcp flags syn / syn,rst tcp option maxseg size 1400-65495 counter packets 0 bytes 0 tcp option maxseg size set rt mtu } } The previous rule is created by NetworkManager to clamp-mss-to-pmtu for the ftth pppoe. This is how I create the bridge and the dac0.28 vlan with NetworkManager: # nmcli con add ifname br28 type bridge con-name br28 ipv4.method manual ipv4.addresses MY_IP/28 connection.zone libvirt # nmcli connection add type vlan con-name dac0.28 ifname dac0.28 vlan.parent dac0 vlan.id 28 ipv4.method disabled ipv6.method disabled master br28 slave-type bridge I also have isc-dhcp-server, wide-dhcpv6-client and radvd running. # nmcli con NAME UUID TYPE DEVICE ftth f370639c-2712-49c2-9749-e39f17102346 pppoe ftth br28 e4d2aad3-ef2d-4ac0-bda5-58471f21655c bridge br28 lo f0327b03-bbc3-4078-8bd1-5225df0ce153 loopback lo vnet1 25ae75cd-1606-4fd7-8213-09f4ef1280c4 tun vnet1 dac0 040e747e-fd7e-41e9-b6a6-ccec9e73c022 ethernet dac0 dac0.100 147c1632-2c60-42f3-a97a-a6733ef69f4c vlan dac0.100 dac0.28 cefb4bf3-dda9-465a-95d0-512ac1294a5b vlan dac0.28 enp1s0 81a44a95-efdc-47e2-9c12-76a0a140ca5a ethernet -- The previous are all dark green expect lo and vnet1 which are light green (externally managed) and enp1s0 which is white (disconnected). The br_netfilter module is not loaded and thus net.bridge.bridge-nf-call-ip6tables, net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-arptables are not even exposed to /proc/sys/net/bridge. I don't have any nwfilter enabled yet, I'd like to get the basics working first. The guest gets an IP (both v4 and v6) and can reach the internet. Unfortunately the guest cannot be reached from the outside: $ ping GUEST_IP PING GUEST_IP (GUEST_IP) 56(84) bytes of data. From FTTH_IP icmp_seq=1 Packet filtered $ ssh GUEST_IP ssh: connect to host GUEST_IP port 22: No route to host Routed networking works fine, but libvirt is in charge to create everything in that case (creating the bridge, assigning the libvirt-routed zone, enabling the libvirt-routed policies, etc) while bridged networking must be configured manually (at least on non-RedHat distros). What's wrong? It looks suspiciously similar to https://bbs.archlinux.org/viewtopic.php?id=274670 Thanks, Niccolo' Belli

1 0

Starting guest VM with default NAT network breaks host routing
by Mathias Gibbens 20 Jun '23

20 Jun '23

I'm running libvirt 9.0.0 on a Debian 12 bookworm host, setting up a Windows 11 guest using qemu-kvm and the default NAT network[1]. When I start the guest VM, it successfully acquires a DHCP lease, and I can ping other hosts on my local network subnet, but something then happens which messes up routing on the *host* machine, resulting in no access to anything beyond my LAN. If I shutdown the VM, network access returns to normal on the host system. I've been unable to figure out how to fix this, and unfortunately the terms are generic enough that Google isn't much help -- most of the results are about issues with NAT setup (which is working), not route configuration of the host. I did find one similar report[2], but there's no reported solution. I feel like this should be a very common use case, so maybe I've just setup something wrong. Since I haven't been able to solve it on my own, I'm hoping someone will have a pointer to get me going in the right direction. Further details are below, and I'm happy to provide anything else that might prove useful. Thanks, Mathias Before starting the guest VM, routes on the host are: > $ ip route > default via 172.20.1.1 dev wlan0 proto dhcp src 172.20.1.110 metric 600 > 169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown > 172.20.1.0/24 dev wlan0 proto kernel scope link src 172.20.1.110 metric 600 > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown A few seconds after the guest finishes booting, the host gets some new routes, and connectivity outside of the local LAN subnet breaks: > $ ip route > 0.0.0.0 dev vnet0 scope link > default dev vnet0 scope link > default via 172.20.1.1 dev wlan0 proto dhcp src 172.20.1.110 metric 600 > 169.254.0.0/16 dev vnet0 proto kernel scope link src 169.254.103.112 > 169.254.0.0/16 dev virbr0 scope link metric 1000 > 172.20.1.0/24 dev wlan0 proto kernel scope link src 172.20.1.110 metric 600 > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 Within the guest, I can ping the NAT gateway (192.168.122.1), the host's IP (172.20.1.110), and another computer on the network (172.20.1.105), so NAT appears to be working correctly: > C:\Users\user>ipconfig /all > > Windows IP Configuration > > Host Name . . . . . . . . . . . . : DESKTOP-LGNEPEC > Primary Dns Suffix . . . . . . . : > Node Type . . . . . . . . . . . . : Hybrid > IP Routing Enabled. . . . . . . . : No > WINS Proxy Enabled. . . . . . . . : No > > Ethernet adapter Ethernet: > > Connection-specific DNS Suffix . : > Description . . . . . . . . . . . : Red Hat VirtIO Ethernet Adapter > Physical Address. . . . . . . . . : 52-54-00-AE-05-B2 > DHCP Enabled. . . . . . . . . . . : Yes > Autoconfiguration Enabled . . . . : Yes > Link-local IPv6 Address . . . . . : fe80::2881:98b7:34b8:fe2%11(Preferred) > IPv4 Address. . . . . . . . . . . : 192.168.122.203(Preferred) > Subnet Mask . . . . . . . . . . . : 255.255.255.0 > Lease Obtained. . . . . . . . . . : Monday, June 19, 2023 14:03:57 > Lease Expires . . . . . . . . . . : Monday, June 19, 2023 15:03:57 > Default Gateway . . . . . . . . . : 192.168.122.1 > DHCP Server . . . . . . . . . . . : 192.168.122.1 > DHCPv6 IAID . . . . . . . . . . . : 340939776 > DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-2C-0E-E6-F4-52-54-00-AE-05-B2 > DNS Servers . . . . . . . . . . . : 192.168.122.1 > NetBIOS over Tcpip. . . . . . . . : Enabled > > C:\Users\user>ping 192.168.122.1 > > Pinging 192.168.122.1 with 32 bytes of data: > Reply from 192.168.122.1: bytes=32 time<1ms TTL=64 > Reply from 192.168.122.1: bytes=32 time<1ms TTL=64 > Reply from 192.168.122.1: bytes=32 time<1ms TTL=64 > Reply from 192.168.122.1: bytes=32 time<1ms TTL=64 > > Ping statistics for 192.168.122.1: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > Approximate round trip times in milli-seconds: > Minimum = 0ms, Maximum = 0ms, Average = 0ms > > C:\Users\user>ping 172.20.1.110 > > Pinging 172.20.1.110 with 32 bytes of data: > Reply from 172.20.1.110: bytes=32 time<1ms TTL=64 > Reply from 172.20.1.110: bytes=32 time<1ms TTL=64 > Reply from 172.20.1.110: bytes=32 time<1ms TTL=64 > Reply from 172.20.1.110: bytes=32 time<1ms TTL=64 > > Ping statistics for 172.20.1.110: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > Approximate round trip times in milli-seconds: > Minimum = 0ms, Maximum = 0ms, Average = 0ms > > C:\Users\user>ping 172.20.1.105 > > Pinging 172.20.1.105 with 32 bytes of data: > Reply from 172.20.1.105: bytes=32 time=49ms TTL=63 > Reply from 172.20.1.105: bytes=32 time=31ms TTL=63 > Reply from 172.20.1.105: bytes=32 time=26ms TTL=63 > Reply from 172.20.1.105: bytes=32 time=26ms TTL=63 > > Ping statistics for 172.20.1.105: > Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), > Approximate round trip times in milli-seconds: > Minimum = 26ms, Maximum = 49ms, Average = 33ms ----- [1] -- Here's the NAT configuration: > <network> > <name>default</name> > <uuid>ff6cd6ed-a8fe-4e50-8852-3c93a169e156</uuid> > <forward mode="nat"> > <nat> > <port start="1024" end="65535"/> > </nat> > </forward> > <bridge name="virbr0" stp="on" delay="0"/> > <mac address="52:54:00:4e:80:30"/> > <ip address="192.168.122.1" netmask="255.255.255.0"> > <dhcp> > <range start="192.168.122.2" end="192.168.122.254"/> > </dhcp> > </ip> > </network> [2] -- https://unix.stackexchange.com/questions/723091/kvm-booting-guest-breaks-ne…

1 0

Changing EFI boot order via API
by Lukas Zapletal 19 Jun '23

19 Jun '23

Hello, I would like to be able to change the boot order of an EFI VM via libvirt API. Specifically, I am looking into configuring a VM to boot over HTTP UEFI Boot or HTTPS UEFI Boot (also available as HTTP IPv4 in the firmware). Is this possible? I cannot find anything relevant in the documentation. Thanks. -- Later, Lukas @lzap Zapletal

2 2