- Users - libvirt List Archives

NVMe drive PCI passthrough and suprise hotplug
by Kalra, Ashish 08 Feb '22

08 Feb '22

[AMD Official Use Only] Hi, I am using Fedora 33, with the following KVM, qemu and libvirt versions: QEMU 5.1.0 libvirt 6.6.0 KVM 5.14.18 We have done pass-through of a PCIe NVMe device to the guest running on FC33 using either virt-manager or virsh and then we do the hot-unplug of the device while it is attached to the guest. The device is no longer seen on the guest hardware device list on virt-manager and then we hotplug the device again and we are able to use it on the Host, but when we try to re-attach it to the guest, we get the following error message: Requested operation is not valid, PCI device 0000:c4::00.0 is in use by driver QEMU, Domain fedora 33. So somehow libvirt still thinks the hot-unplugged device is attached. Tracing the flow of hot un-plug event from guest to host : ->Guest pcie hotplug support detected the NVMe driver unplug (from guest kernel logs): pciehp: Slot (0-6): Attention button pressed pciehp: Slot (0-6): Powering off due to to button press. -> Also looks like the guest notified Host/KVM (from host kernel logs): pcieport: 0000:c4:0000.0: pciehp: Slot (208): Card not present -> Correspondingly, vfio-pci module notified Qemu : vfio-pci: 0000:c4:0000.0: Relaying device request to user (#0) -> Then the un-plugged device reset is done. vfio-pci: vfio_bar_restore: reset recovery - restoring BARs pci 0000:c4:00.0: Removing from iommu group 105. -> Next tried to verify if libvirt detected the DELETED_DRIVE event from qemu. Running SystemTap script to capture events between qemu and libvirt : stap examples/systemtap/qemu-monitor.stp When the NVMe drive is attached to VM the following log output is seen from SystemTap: execute "device-add", driver: "vfio-pci", host: "0000:c4:00.0", id: "hostdev0", bus: "pci.7", addr: "0". When we hot-unplug the NVMe drive, the following log output is seen from SystemTap: event: DEVICE_DELETED, device: "hostdev0", path: "/machine/peripheral/hostdev0". So it looks like that qemu sent the "DEVICE_DELTED" event to libvirt, but libvirt has still not removed the attached device from its bookeeping list. I understand there is already a thread from 20202, discussing a similar issue : https://www.spinics.net/linux/fedora/libvirt-users/msg12590.html But I am not sure if there is any fix/support added for this recently. Looking for any feedback related to above and PCI device passthrough and hotplug support. Thanks, Ashish

2 2

udmabuf error with libvirt + QEMU
by M, Shivakumar 01 Feb '22

01 Feb '22

Hi, We are seeing an issue with udambuf, where it says "open /dev/udmabuf: No such file or directory " even if the device exits. This issue particularly we are seeing with libvirt. When we run the QEMU args on the command line, everything works as expected. It seems to be some permission issue when we use the Libvirt, please help us on resolving this. [cid:image001.png@01D817A5.0151F260] Libvirt Version: 6.0.0 Host: Ubuntu 21.04 QEMU commands: qemu-system-x86_64 \ -enable-kvm \ -smp cores=4,threads=2,sockets=1 \ -m 4096 \ -name win-vm-0 \ -cpu host \ -drive id=windows_drive,if=virtio,file=/win10_enterprise.qcow2,format=qcow2,cache=none \ -device vfio-pci,host=0000:00:02.2 \ -device e1000,netdev=net0,mac=00:a1:00:00:00:10 \ -netdev user,id=net0,ipv6=off,hostfwd=tcp::1111-:22,hostfwd=tcp::3389-:3389 \ -monitor tcp:127.0.0.1:11111,server,nowait \ -rtc base=localtime \ -usb \ -device usb-tablet \ -device usb-kbd \ -device usb-mouse \ -device virtio-vga,blob=true \ -display gtk,gl=on \ -object memory-backend-memfd,id=mem1,hugetlb=off,size=4096M \ -machine memory-backend=mem1 Libvirt XML: <?xml version="1.0"?> <domain xmlns:qemu=http://libvirt.org/schemas/domain/qemu/1.0 type="kvm"> <name>win-vm-0</name> <memory unit="KiB">4194304</memory> <currentMemory unit="KiB">4194304</currentMemory> <memoryBacking supported="yes"> <source type="memfd"/> </memoryBacking> <vcpu>6</vcpu> <os> <type arch="x86_64" machine="pc">hvm</type> <boot dev="hd"/> <boot dev="cdrom"/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state="on"/> <vapic state="on"/> <spinlocks state="on" retries="8191"/> </hyperv> <vmport state="off"/> </features> <cpu mode="host-model"> <model fallback="forbid"/> </cpu> <clock offset="utc"> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2"/> <source file="/win10_enterprise.qcow2" index="1"/> <backingStore/> <target dev="sda" bus="sata"/> <alias name="sata0-0-0"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk> <interface type="bridge"> <source bridge="br0"/> <model type="virtio"/> </interface> <serial type="pty"> <target port="0"/> <target type="serial" port="0"> </target> <alias name="serial0"/> </serial> <console type="pty"> <target type="serial" port="0"/> <alias name="serial0"/> </console> <hostdev mode="subsystem" type="pci" managed="yes"> <driver name="vfio"/> <source> <address domain="0x0000" bus="0x00" slot="0x02" function="0x02"/> </source> </hostdev> </devices> <qemu:commandline> <qemu:arg value="-device"/> <qemu:arg value="virtio-vga,blob=true"/> <qemu:arg value="-display"/> <qemu:arg value="gtk,gl=on"/> <qemu:env name="DISPLAY" value=":1.0"/> </qemu:commandline> </domain> Thanks, Shiv

3 3

frequent network collapse possibly due to bridging
by Hakan E. Duran 30 Jan '22

30 Jan '22

Hi, I would like some help to troubleshoot the problem I have been having lately with my VM host, which contains 5 VMs, one of which is for pi-hole, unbound services. It has been a relatively common occurrence in the last few weeks for me to find that the host machine has lost its network when I get back home from work. Restoring the VM/VMs do not fix the problem, the host needs to be restarted for a fix, otherwise there is both loss of name resolution, as well as an internet connection; I cannot ping even IPs such as 8.8.8.8. Since I use the pi-hole VM as the DNS server for my LAN, this means that my whole LAN gets disconnected from internet, until the host machine is rebooted. The host machine has a little complicated network setup: the two gigabit connections are bonded and bridged to the VMs; however this set up has been serving me so well for several years now. The problem, on the other hand, appeared a few weeks ago. This doesn't happen every day but often enough to be annoying and disruptive for my family. My question is, how can I troubleshoot this problem and figure out whether it is truly due to network bridging somehow collapsing or not? I tried to find some log files but all I could find were the /var/log/libvirt/qemu/$VM files, and the particular log file for the pi-hole VM reported the following lines; however, I am not sure if they are associated with a real crash or just due to shutting down and restarting the host (please excuse the word-wrapping): char device redirected to /dev/pts/2 (label charserial0) qxl_send_events: spice-server bug: guest stopped, ignoring 2022-01-20T23:41:17.012445Z qemu-system-x86_64: terminating on signal 15 from pid 1 (/sbin/init) 2022-01-20 23:41:17.716+0000: shutting down, reason=crashed 2022-01-20 23:42:46.059+0000: starting up libvirt version: 7.10.0, qemu version: 6.2.0, kernel: 5.10.89-1-MANJARO, hostname: -redacted- Please excuse my ignorance but is there a way to restart the networking without rebooting the host machine? This will not solve my problem since I won't be able to reach to the host remotely if the networking is down. The real solution would be preventing these network crashes and the first step in that would be effective troubleshooting in my opinion. Any input/guidance will be greatly appreciated. I can provide more info about my host/VM(s) if the above is not adequate. Thanks, Hakan Duran

4 6

Backend memory object creation - query
by M, Shivakumar 27 Jan '22

27 Jan '22

Hello, For our use-case with Libvirt we want to create the Memory backend object , Expected QEMU args would be -object memory-backend-memfd,id=mem1,size=4096M Request you to please help us to specify this arg in the libvirt XML. Thanks, Shiv

2 1

cpuset.mems - Device or resource busy
by lejeczek 26 Jan '22

26 Jan '22

Hi guys On Centos 9 I get these in journal: ... libvirt version: 8.0.0, package: 1.el9 (builder(a)centos.org, 2022-01-14-15:00:06, ) hostname: dzien.mine.private Unable to write to '/sys/fs/cgroup/machine.slice/machine-qemu\x2d4\x2dubusrv2.scope/libvirt/cpuset.mems': Device or resource busy ... each time I do: -> virt-admin daemon-log-outputs There is more VMs running on the host but it's only these Ubuntu VMs Libvirt is not happy about. Any and all suggestions very appreciated. many thanks, L

1 0

Best practices for retrying QEMU P2P migrations
by Raphael Norwitz 24 Jan '22

24 Jan '22

Heya, We’ve recently hit a number of failures on QEMU P2P live migrations which appear to be caused by transient networking disconnects at different points in the migration process. We would like to implement smarter retry logic in our control plane to ensure such issues don’t stall critical workflows. On the other hand, we cannot blindly retry every failed migration because doing so greatly lengthens the time to fail high level autmation when there is a real problem. Are there currently any generally understood best practices for retrying migrations from a control plane perspective? Ideally we would decide whether or not to retry based on error codes, but especially in the QEMU P2P migration path many generic codes are returned. For example, see [1] where we attempted to improve an error code for a likely retry-able set of failure cases. [1] https://listman.redhat.com/archives/libvir-list/2022-January/msg00217.html Thanks, Raphael

1 0

What does the positional parameters of "virsh backup-begin" actually do?
by Ahmad Ismail 15 Jan '22

15 Jan '22

Normally when I backup a kvm machine I shutdown the machine then run: virsh list -all virsh shutdown Ubuntu18 virsh dumpxml Ubuntu18 > /MyBackup/Ubuntu18.xml cp /var/lib/libvirt/images/Ubuntu18.qcow2 /MyBackup/Ubuntu18.qcow2 However, I found a new sub-command. The help file says: % virsh backup-begin --help NAME backup-begin - Start a disk backup of a live domain SYNOPSIS backup-begin <domain> [--backupxml <string>] [--checkpointxml <string>] [--reuse-external] DESCRIPTION Use XML to start a full or incremental disk backup of a live domain, optionally creating a checkpoint OPTIONS [--domain] <string> domain name, id or uuid --backupxml <string> domain backup XML --checkpointxml <string> domain checkpoint XML --reuse-external reuse files provided by caller The problem with this help is, it is not clear enough. I understand that I should use virsh backup-begin vm1 to backup a live kvm machine. However, this command only create .qcow2 files. What about the .xml file. What does --backupxml , --checkpointxml & --reuse-external actually do? When should I use them?

2 2

interface hotplug q35 machine type
by Miguel Duarte de Mora Barroso 13 Jan '22

13 Jan '22

Hello, I see that as part of libvirt's documentation [0] the q35 machine type will feature at most 1 hotplugged PCIe device - default - and users must prepare in advance according to their expectations of how many ifaces will be hotplugged: """ Slots on the pcie-root controller do not support hotplug, so the device will be hotplugged into the pcie-root-port controller. If you plan to hotplug more than a single PCI Express device, you should add a suitable number of pcie-root-port controllers when defining the guest: for example, add ```xml <controller type='pci' model='pcie-root'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> ``` if you expect to hotplug up to three PCI Express devices, either emulated or assigned from the host. """ Is there any alternative to this ? For our use case, I'm considering mimicking Openstack's implementation - [1] - and expose a knob that indicates what is the number of PCIe root ports to be used upon the domain definition. I wonder how open would the community be to having a machine type alias that would provide a "better" default - in the sense that it would have more root port controllers. [0] - https://libvirt.org/pci-hotplug.html#x86_64-q35 [1] - https://blueprints.launchpad.net/nova/+spec/configure-amount-of-pcie-ports

2 1

DMARC & similar @admins
by lejeczek 12 Jan '22

12 Jan '22

Hi guys My memory fools me? - I remembered this list had DMAR+other_bits sorted, so users like myself, on Yahoo, did not loose their own emails. regards, L

2 3

'migrate' says it worked but in reality it did not - centOS 9
by lejeczek 12 Jan '22

12 Jan '22

Hi guys. I have a peculiar situation where between boxes: C->A -> $ virsh migrate --unsafe --live c8kubermaster1 qemu+ssh://10.1.1.99/system -> $ echo $? 0 but above does _not_ happen, instead!! VM was stopped in started, but _not_ migrated LIVE A->C -> $ virsh migrate --unsafe --live c8kubermaster1 qemu+ssh://10.1.1.100/system -> $ echo $? 0 indeed VM migrates live. box A & C have virtually identical OS stack, HW difference is: C = Ryzen 5 5600G A = Ryzen 5 3600 domain XML snippet where I think it matters: ... </metadata> <memory unit='GiB'>4</memory> <currentMemory unit='GiB'>4</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC-IBPB</model> <feature policy='require' name='ibpb'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='virt-ssbd'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> ... Initially I submitted a BZ against 'PCS' but continued to filled with it and I find 'libvirt' might be the culprit(also?) here. There is not much in logs, certainly nothing (with default verbosity) in virtqemud.service Is it that VM gets migrated but then is restarted on 'migrate_to' host? if so then why? How to start troubleshooting such 'monstrosity'? - all suggestions appreciated. many thanks, L.

2 4