- Users - libvirt List Archives

Number of the max supported VFs are different
by Yalan Zhang 08 Mar '22

08 Mar '22

Hi there, I have an Intel X520 network card, and I find the max supported VFs are different. Please check below outputs: # lspci -vvv -s 04:00.0 04:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) ... Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- ** Initial VFs: 64, Total VFs: 64**, Number of VFs: 0, Function Dependency Link: 00 VF offset: 128, stride: 2, Device ID: 10ed Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 0000000094400000 (64-bit, prefetchable) Region 3: Memory at 0000000094500000 (64-bit, prefetchable) VF Migration: offset: 00000000, BIR: 0 Kernel driver in use: ixgbe Kernel modules: ixgbe # cat /sys/class/net/enp4s0f0/device/sriov_totalvfs 63 # echo 64 > /sys/class/net/enp4s0f0/device/sriov_numvfs -bash: echo: write error: Numerical result out of range # echo 63 > /sys/class/net/enp4s0f0/device/sriov_numvfs # cat /sys/class/net/enp4s0f0/device/sriov_numvfs 63 The lspci command says the Total VFs supported is 64, while in the file "sriov_totalvfs" says it's 63. And the sriov_numvfs file will take precedence. Why are the numbers different? Just a little curious. Thank you! ------- Best Regards, Yalan Zhang IRC: yalzhang

2 2

libvirtd file descriptors leaking
by Дмитрий 01 Mar '22

01 Mar '22

Hello, ## SETUP CentOS 8 Stream libguestfs-tools.noarch 1:1.40.2-24.el8.plesk libvirt.x86_64 7.6.0-6.el8s ## ISSUE REPRODUCTION lsof -p $(cat /run/libvirtd.pid) | wc -l # here we have N open file descriptors by libvirtd guestfish --ro -d domain1 run # here we have N+1 open file descriptors by libvirtd Ctrl+D # and we still have N+1 open file descriptors by libvirtd virt-df -d domain1 # after completion we still having N++ open file descriptors by libvirtd To fix this issue i am force to restart libvirtd

3 7

QEMU args generation order with libvirt
by M, Shivakumar 28 Feb '22

28 Feb '22

Hello, We are trying to bring up the Android VM with Libvirt, as part of this exercise it is advised to keep Replay Protected Memory Block (RPMB) device as the first virtio device in the QEMU arg list, as the secure storage daemon in Andriod side communicates with dev/vport0p1 for RPMB usage. Is there any way we can make a qemu commands generation in particular order with Libvirt XML ? Thanks, Shivakumar M

2 1

virtio 3d acceleration - centOS 9 - ?
by lejeczek 27 Feb '22

27 Feb '22

Hi guys Would anybody know if RHEL & all involved gangs plan or think to have c9, when is GA, offer/support 3d virtio? many thanks, L.

1 0

networking question
by Natxo Asenjo 26 Feb '22

26 Feb '22

hi, I have an issue with one host at a customer's site. I think this cannot work, but I would like to ask you just in case I am confused. host: eno1: 172.20.10.x/24 management interface gw 172.20.10.254 bridge-service: 0.0.0.0/24 tun0: openvpn tunnel to external data center internal-bridge: x.x.x.x/28 ; routed subnet that goes to openvpn tun0 on vm: eth0: x.x.x.x/28 on internal-bridge (default gw) eth1: 172.20.10.x/24 bridge-service gw 172.20.10.254 (same as eno1) Connectivity to and from openvpn (from and to datacenter) is perfect. All vms are directly reachable from our management services, no natting. >From hypervisor I can ping the gw, from vm I cannot ping 172.20.10.254. My gut feeling is that this cannot work because traffic for the hypervisor for subnet 172.20.10.x/24 flows through eno1, but for vm through the bridge-loggin interface. So that cannot work. Should we just ask the customer to give us different subnets for the host and the vm? TIA. -- regards, Natxo

3 2

domblkinfo->allocation is not equals to domstats.1->allocation for disk with <slices> settings
by Zhen Tang 24 Feb '22

24 Feb '22

Hi, I did some testing about disk with <slice> setting and found that the value of domblkinfo->allocation is not same with domstats.1->allocation for disk with <slices> Env: libvirt-8.0.0-4.el9 qemu-kvm-6.2.0-9.el9 Step: 1. prepare a image # qemu-img create /car/lib/libvirt/images/disk-raw -f raw 100M -o preallocation=full 2. start a guest and attach the disk ➜ ~ cat disk.xml <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native' copy_on_read='off' discard='ignore' detect_zeroes='on'/> <source file='/var/lib/libvirt/images/disk-raw' index='1'> <slices> <slice type='storage' offset='0' size='104857600'/> </slices> </source> <backingStore/> <target dev='sdb' bus='scsi'/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> <alias name='ua-slices'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> 3. Write some data to the slices disk in vm (in vm) dd if=/dev/zero of=/dev/sdb bs=10M count=1 4. check *domblkinfo* and *domstats* ➜ ~ virsh domstats rhel9.0-1 --block|grep -Ei 'block.1.(allocation|capacity|physical)' *block.1.allocation=0* block.1.capacity=104857600 block.1.physical=104865792 ➜ ~ ➜ ~ virsh domblkinfo rhel9.0-1 sdb|grep -Ei '(allocation|capacity|physical)' Capacity: 104857600 *Allocation: 104865792* Physical: 104857600 Are these 'allocation' values expected to be equal to each other? Thanks, Zhen Tang

1 0

Re: Hugepages -- "Memory backend 'pc.ram' not found"
by Michal Prívozník 23 Feb '22

23 Feb '22

On 2/21/22 17:12, Charles Polisher wrote: Hey, please the list on CC for benefit of others, e.g. when somebody runs into the same problem they can find the discussion in the archive. > On 2/21/22 01:54, Michal Prívozník wrote: > >> On 2/20/22 04:07, Charles Polisher wrote: >>> Hello, >>> >>> After defining hugepages, as documented at >>> https://libvirt.org/formatdomain.html#memory-backing , >>> when I start the guest, I get a dialogue >>> box that says: >>> >>> Error starting domain: internal error: qemu unexpectedly >>> closed the monitor: 2022-02-20T01:10:36.520955Z >>> qemu-system-x86_64: Memory backend 'pc.ram' not found >>> Traceback (most recent call last): >>> File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, >>> in cb_wrapper >>> callback(asyncjob, *args, **kwargs) >>> File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, >>> in tmpcb >>> callback(*args, **kwargs) >>> File >>> "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, >>> in newfn >>> ret = fn(self, *args, **kwargs) >>> File "/usr/share/virt-manager/virtManager/object/domain.py", line >>> 1329, in startup >>> self._backend.create() >>> File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1353, >>> in create >>> raise libvirtError('virDomainCreate() failed' >>> >>> After backing out changes, guest starts normally. >>> I searched online for the error message, but found nothing useful. >>> The hypervisor is running libvirtd (libvirt) 7.8.0 and QEMU emulator >>> version 6.1.0, >>> both build from source. I've got plenty of hugepages available. >>> The domain's XML definition is attached. >> Hey, can you share your domain XML and the generated cmd line? The >> latter should be found in /var/log/libvirt/qemu/$domain.log >> >> Thanks, >> Michal > > Thanks for your reply. As requested, the domain XML: > > <domain type="kvm"> > <name>slacky-0</name> > <uuid>4a67eb39-9b92-8b8a-97ba-7e1250d56b07</uuid> > <title>slacky-0</title> > <description>elided</description> > <memory unit="KiB">4194304</memory> > <currentMemory unit="KiB">4194304</currentMemory> > <memoryBacking> > <hugepages> > <page size="4194304" unit="KiB"/> This does not look correct. This is on x86_64 and there is no such size for hugepages, only 2MiB and 1GiB. > </hugepages> > </memoryBacking> > <vcpu placement="static">2</vcpu> > <os> > <type arch="x86_64" machine="pc-i440fx-5.1">hvm</type> > <bootmenu enable="no"/> > </os> > <features> > <acpi/> > <apic/> > <pae/> > </features> > <cpu mode="custom" match="exact" check="none"> > <model fallback="forbid">kvm64</model> > </cpu> > <clock offset="utc"/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <devices> > <emulator>/usr/bin/qemu-system-x86_64</emulator> > <disk type="file" device="cdrom"> > <driver name="qemu" type="raw"/> > <target dev="hdc" bus="ide"/> > <readonly/> > <address type="drive" controller="0" bus="1" target="0" > unit="0"/> > </disk> > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" cache="writethrough"/> > <source file="/mnt/nvme1/VIRTUAL_MACHINES/slacky-0.qcow2"/> > <target dev="vda" bus="virtio"/> > <boot order="1"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x09" > function="0x0"/> > </disk> > <controller type="usb" index="0" model="ich9-ehci1"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x05" > function="0x7"/> > </controller> > <controller type="usb" index="0" model="ich9-uhci1"> > <master startport="0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x05" > function="0x0" multifunction="on"/> > </controller> > <controller type="usb" index="0" model="ich9-uhci2"> > <master startport="2"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x05" > function="0x1"/> > </controller> > <controller type="usb" index="0" model="ich9-uhci3"> > <master startport="4"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x05" > function="0x2"/> > </controller> > <controller type="ide" index="0"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x01" > function="0x1"/> > </controller> > <controller type="virtio-serial" index="0"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x07" > function="0x0"/> > </controller> > <controller type="scsi" index="0" model="virtio-scsi"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x08" > function="0x0"/> > </controller> > <controller type="pci" index="0" model="pci-root"/> > <interface type="network"> > <mac address="52:54:00:c3:93:40"/> > <source network="default"/> > <model type="virtio"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" > function="0x0"/> > </interface> > <serial type="file"> > <source path="/tmp/myconsoleoutput.txt"/> > <target type="isa-serial" port="0"> > <model name="isa-serial"/> > </target> > </serial> > <console type="file"> > <source path="/tmp/myconsoleoutput.txt"/> > <target type="serial" port="0"/> > </console> > <input type="tablet" bus="usb"> > <address type="usb" bus="0" port="1"/> > </input> > <input type="mouse" bus="ps2"/> > <input type="keyboard" bus="ps2"/> > <graphics type="spice" autoport="yes" listen="127.0.0.1"> > <listen type="address" address="127.0.0.1"/> > </graphics> > <sound model="ich9"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x04" > function="0x0"/> > </sound> > <audio id="1" type="spice"/> > <video> > <model type="qxl" ram="65536" vram="65536" vgamem="16384" > heads="1" primary="yes"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0"/> > </video> > <memballoon model="virtio"> > <address type="pci" domain="0x0000" bus="0x00" slot="0x06" > function="0x0"/> > </memballoon> > </devices> > </domain> > > > And the guest log with the generated command line: > > 2022-02-20 01:13:12.985+0000: starting up libvirt version: 7.8.0, > qemu version: 6.1.0, kernel: 5.15.19, hostname: godzilla.peecee3.com > LC_ALL=C \ > PATH=/sbin:/usr/sbin:/bin:/usr/bin \ > HOME=/var/lib/libvirt/qemu/domain-34-slacky-0 \ > XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-34-slacky-0/.local/share \ > XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-34-slacky-0/.cache \ > XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-34-slacky-0/.config \ > /usr/bin/qemu-system-x86_64 \ > -name guest=slacky-0,process=qemu:slacky-0,debug-threads=on \ > -S \ > -object > '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-34-slacky-0/master-key.aes"}' > \ > -machine > pc-i440fx-5.1,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \ So this instructs qemu to use a memory device wih id='pc.ram' as the default/generic memory for the guest.. > -cpu kvm64 \ > -m 4096 \ > -overcommit mem-lock=off \ .. but we never generate such device. Here libvirt should have generated -object memory-backend-file,id=pc.ram,path=/hugepages/... And I think I know why. Let me post a patch. > -smp 2,sockets=2,cores=1,threads=1 \ > -uuid 4a67eb39-9b92-8b8a-97ba-7e1250d56b07 \ > -no-user-config \ > -nodefaults \ > -chardev socket,id=charmonitor,fd=34,server=on,wait=off \ > -mon chardev=charmonitor,id=monitor,mode=control \ > -rtc base=utc \ > -no-shutdown \ > -boot menu=off,strict=on \ > -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \ > -device > ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 > \ > -device > ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \ > -device > ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \ > -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x8 \ > -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 \ > -device ide-cd,bus=ide.1,unit=0,id=ide0-1-0 \ > -blockdev > '{"driver":"file","filename":"/mnt/nvme1/VIRTUAL_MACHINES/slacky-0.qcow2","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' > \ > -blockdev > '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' > \ > -device > virtio-blk-pci,bus=pci.0,addr=0x9,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=off > \ > -netdev tap,fd=58,id=hostnet0,vhost=on,vhostfd=60 \ > -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c3:93:40,bus=pci.0,addr=0x3 > \ > -add-fd set=3,fd=62 \ > -chardev file,id=charserial0,path=/dev/fdset/3,append=on \ > -device isa-serial,chardev=charserial0,id=serial0 \ > -device usb-tablet,id=input0,bus=usb.0,port=1 \ > -audiodev id=audio1,driver=spice \ > -spice > port=5901,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on \ > -device > qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 > \ > -device ich9-intel-hda,id=sound0,bus=pci.0,addr=0x4 \ > -device > hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=audio1 \ > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \ > -msg timestamp=on > 2022-02-20T01:13:13.136602Z qemu-system-x86_64: Memory backend > 'pc.ram' not found > 2022-02-20 01:13:13.186+0000: shutting down, reason=failed > > Again, thank you! Michal

2 2

help
by admin＠foundryserver.com 23 Feb '22

23 Feb '22

help

1 0

random but often VMs' - BUG: soft lockup
by lejeczek 22 Feb '22

22 Feb '22

Hi guys I see this often(perhaps once a day) on VMs: ... watchdog: BUG: soft -lockup - CPU#1 stuck for xxXXxxs! [swapper/1:0] At the moment I cannot say I see a pattern - perhaps not spotted one yet - and this can happen to any VM between any one of the three hosts, which hosts are CentOS 9 on AMD Ryzen 5600G hardware. VMs are mostly CentOS 8 but a few Ubuntus too. Anybody would know what that might be, is, a symptom of? But most importantly how to dig in to troubleshoot this?(when this happen then such affected VM I can only hard reset) many thanks, L.

2 1

Hugepages -- "Memory backend 'pc.ram' not found"
by Charles Polisher 21 Feb '22

21 Feb '22

Hello, After defining hugepages, as documented at https://libvirt.org/formatdomain.html#memory-backing , when I start the guest, I get a dialogue box that says: Error starting domain: internal error: qemu unexpectedly closed the monitor: 2022-02-20T01:10:36.520955Z qemu-system-x86_64: Memory backend 'pc.ram' not found Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/object/domain.py", line 1329, in startup self._backend.create() File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1353, in create raise libvirtError('virDomainCreate() failed' After backing out changes, guest starts normally. I searched online for the error message, but found nothing useful. The hypervisor is running libvirtd (libvirt) 7.8.0 and QEMU emulator version 6.1.0, both build from source. I've got plenty of hugepages available. The domain's XML definition is attached. Any ideas where to look next? Thanks, -- Charles Polisher

2 1

libvirt-python for alpine linux
by Tom Ammon 21 Feb '22

21 Feb '22

Hi everybody, Is there a libvirt-python package for alpine linux? I'm looking to containerize an app that uses libvirt-python, using alpine linux, but all I found that looked similar to libvirt-python was py-libvirt ( https://pkgs.alpinelinux.org/package/v3.3/main/x86/py-libvirt) and it looks pretty old. Tom -- ----------------------------------------------------------------------------- Tom Ammon M: (737) 400-9042 thomasammon(a)gmail.com -----------------------------------------------------------------------------

2 2

SSH VM from outside, but not from host
by Wolf 17 Feb '22

17 Feb '22

Hi! 1) I have two network ports on my server. - eno1 has the IP: XX1.XX1.XX1.150 - bridge0 has the IP: XX2.XX2.XX2.100 and has the interface member: port eno2. eno2 is not set up with an IP address. 2) The host runs on IP: XX1.XX1.XX1.150 3) A VM uses the bridge: bridge0, and has the IP: XX2.XX2.XX2.100 I have a problem with this setup: I can ssh the VM on XX2.XX2.XX2.100 from outside, but from the host, XX1.XX1.XX1.150, I can't ssh the VM on XX2.XX2.XX2.100. Have I set up this wrong or is it something I can do to solve this? Thanks! Wolf

4 6

Public IP on virtual machine network issue
by Marcin Groszek 16 Feb '22

16 Feb '22

I have been struggling with this for weeks and I was unable to find an answer on line. Perhaps someone here can help me. Oracle linux 8 running virtualization: hardware node has a public IP address on interface bridge0 and physical eno1 is a member of the bridge0 a virtual OS has interface bridged to lan and source is bridge0, Ip address of virtual OS is also a public from same class as the hardware node. I can route in and out of virtual, I can ping from hardware node to virtual and vice versa, so the routing works as it should, sort of. When I try tracepath or traceroute from outside to virtual I get !H on last hup same result when I try to do the same form hardware node to virtual I get !H Also, when I telnet (TCP) to a specific port on virtual where I have a daemon LISTENING OR NOT I get: No route to host. Same experiment works just fine for ssh port. Firewalld is not running, and I just have very basic iptables rules like allowing external address block to ssh to hardware node and to virtual dropping connections from all other sources This issue presented it self when I attempted to setup a galera node on virtual and ports 4567 is responding but 4568 and 4444 are not, but the daemons are running and I can clearly see lsoft showing "LISTENING" I capture the traffic and the tcp as well as udp are getting to the virtual. Is there a preconfigured netfiltering that I am not aware of? What am I missing? -- Best Regards: Marcin Groszek Business Voip Resource. http://www.voipplus.net

4 6

libvirtd daemon missing in LFS
by Sai Kiran Kumar Reddy 15 Feb '22

15 Feb '22

Hi, I am Sai Kiran. I am trying to build libvirt from source on my Linux From Scratch(LFS) system. I have successfully installed libvirt and its dependencies. When I start virt-manager, there is a prompt in GUI saying that "libvirtd service is not installed". I also do not see any libvirtd in my system. I would like to install the libvirtd service(build from source). But I am not able to find the source code for it and also, I am not sure about the build process for the daemon. Could you please help me out here. Thanks in advance for your time and support. -- Regards, Sai Kiran.

3 7

qemu+ssh connections to a remote libvirt fail as ssh banner configured
by Yalan Zhang 14 Feb '22

14 Feb '22

Hi there, I have a system configured with ssh login banner like as below: # cat ~/.bashrc ... echo "=================================================================================" echo "====== This machine is occupied by xxx for testing now. If you are about to use it, contact xxx first ======" echo "=================================================================================" It works as expected that whenever someone logs into this system by ssh, he/she will see this warning message. But it seems such settings will impact a virsh client connection with ssh, when I try to connect the libvirt daemon on this system, it will error out : # virsh -c qemu+ssh://${my_host}/system list --all root@${my_host}'s password: error: failed to connect to the hypervisor error: packet 1027423545 bytes received from server too large, want 33554432 I have searched and found some related explanations[1], and [2] says "The virsh man page doesn't mention ssh, so it sounds like the file /usr/share/doc/libvirt-doc/remote.html shipped with libvirt-doc could use a patch mentioning this." But I can not find anything about this currently on file:///usr/share/doc/libvirt-docs/html/remote.html. Could we have this documented for reference with all the possibilities? Thank you! [1] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/868753/comments/17 [2] https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/868753/comments/14 ------- Best Regards, Yalan Zhang IRC: yalzhang

4 6

NVMe drive PCI passthrough and suprise hotplug
by Kalra, Ashish 08 Feb '22

08 Feb '22

[AMD Official Use Only] Hi, I am using Fedora 33, with the following KVM, qemu and libvirt versions: QEMU 5.1.0 libvirt 6.6.0 KVM 5.14.18 We have done pass-through of a PCIe NVMe device to the guest running on FC33 using either virt-manager or virsh and then we do the hot-unplug of the device while it is attached to the guest. The device is no longer seen on the guest hardware device list on virt-manager and then we hotplug the device again and we are able to use it on the Host, but when we try to re-attach it to the guest, we get the following error message: Requested operation is not valid, PCI device 0000:c4::00.0 is in use by driver QEMU, Domain fedora 33. So somehow libvirt still thinks the hot-unplugged device is attached. Tracing the flow of hot un-plug event from guest to host : ->Guest pcie hotplug support detected the NVMe driver unplug (from guest kernel logs): pciehp: Slot (0-6): Attention button pressed pciehp: Slot (0-6): Powering off due to to button press. -> Also looks like the guest notified Host/KVM (from host kernel logs): pcieport: 0000:c4:0000.0: pciehp: Slot (208): Card not present -> Correspondingly, vfio-pci module notified Qemu : vfio-pci: 0000:c4:0000.0: Relaying device request to user (#0) -> Then the un-plugged device reset is done. vfio-pci: vfio_bar_restore: reset recovery - restoring BARs pci 0000:c4:00.0: Removing from iommu group 105. -> Next tried to verify if libvirt detected the DELETED_DRIVE event from qemu. Running SystemTap script to capture events between qemu and libvirt : stap examples/systemtap/qemu-monitor.stp When the NVMe drive is attached to VM the following log output is seen from SystemTap: execute "device-add", driver: "vfio-pci", host: "0000:c4:00.0", id: "hostdev0", bus: "pci.7", addr: "0". When we hot-unplug the NVMe drive, the following log output is seen from SystemTap: event: DEVICE_DELETED, device: "hostdev0", path: "/machine/peripheral/hostdev0". So it looks like that qemu sent the "DEVICE_DELTED" event to libvirt, but libvirt has still not removed the attached device from its bookeeping list. I understand there is already a thread from 20202, discussing a similar issue : https://www.spinics.net/linux/fedora/libvirt-users/msg12590.html But I am not sure if there is any fix/support added for this recently. Looking for any feedback related to above and PCI device passthrough and hotplug support. Thanks, Ashish

2 2

udmabuf error with libvirt + QEMU
by M, Shivakumar 02 Feb '22

02 Feb '22

Hi, We are seeing an issue with udambuf, where it says "open /dev/udmabuf: No such file or directory " even if the device exits. This issue particularly we are seeing with libvirt. When we run the QEMU args on the command line, everything works as expected. It seems to be some permission issue when we use the Libvirt, please help us on resolving this. [cid:image001.png@01D817A5.0151F260] Libvirt Version: 6.0.0 Host: Ubuntu 21.04 QEMU commands: qemu-system-x86_64 \ -enable-kvm \ -smp cores=4,threads=2,sockets=1 \ -m 4096 \ -name win-vm-0 \ -cpu host \ -drive id=windows_drive,if=virtio,file=/win10_enterprise.qcow2,format=qcow2,cache=none \ -device vfio-pci,host=0000:00:02.2 \ -device e1000,netdev=net0,mac=00:a1:00:00:00:10 \ -netdev user,id=net0,ipv6=off,hostfwd=tcp::1111-:22,hostfwd=tcp::3389-:3389 \ -monitor tcp:127.0.0.1:11111,server,nowait \ -rtc base=localtime \ -usb \ -device usb-tablet \ -device usb-kbd \ -device usb-mouse \ -device virtio-vga,blob=true \ -display gtk,gl=on \ -object memory-backend-memfd,id=mem1,hugetlb=off,size=4096M \ -machine memory-backend=mem1 Libvirt XML: <?xml version="1.0"?> <domain xmlns:qemu=http://libvirt.org/schemas/domain/qemu/1.0 type="kvm"> <name>win-vm-0</name> <memory unit="KiB">4194304</memory> <currentMemory unit="KiB">4194304</currentMemory> <memoryBacking supported="yes"> <source type="memfd"/> </memoryBacking> <vcpu>6</vcpu> <os> <type arch="x86_64" machine="pc">hvm</type> <boot dev="hd"/> <boot dev="cdrom"/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state="on"/> <vapic state="on"/> <spinlocks state="on" retries="8191"/> </hyperv> <vmport state="off"/> </features> <cpu mode="host-model"> <model fallback="forbid"/> </cpu> <clock offset="utc"> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2"/> <source file="/win10_enterprise.qcow2" index="1"/> <backingStore/> <target dev="sda" bus="sata"/> <alias name="sata0-0-0"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk> <interface type="bridge"> <source bridge="br0"/> <model type="virtio"/> </interface> <serial type="pty"> <target port="0"/> <target type="serial" port="0"> </target> <alias name="serial0"/> </serial> <console type="pty"> <target type="serial" port="0"/> <alias name="serial0"/> </console> <hostdev mode="subsystem" type="pci" managed="yes"> <driver name="vfio"/> <source> <address domain="0x0000" bus="0x00" slot="0x02" function="0x02"/> </source> </hostdev> </devices> <qemu:commandline> <qemu:arg value="-device"/> <qemu:arg value="virtio-vga,blob=true"/> <qemu:arg value="-display"/> <qemu:arg value="gtk,gl=on"/> <qemu:env name="DISPLAY" value=":1.0"/> </qemu:commandline> </domain> Thanks, Shiv

3 3

frequent network collapse possibly due to bridging
by Hakan E. Duran 30 Jan '22

30 Jan '22

Hi, I would like some help to troubleshoot the problem I have been having lately with my VM host, which contains 5 VMs, one of which is for pi-hole, unbound services. It has been a relatively common occurrence in the last few weeks for me to find that the host machine has lost its network when I get back home from work. Restoring the VM/VMs do not fix the problem, the host needs to be restarted for a fix, otherwise there is both loss of name resolution, as well as an internet connection; I cannot ping even IPs such as 8.8.8.8. Since I use the pi-hole VM as the DNS server for my LAN, this means that my whole LAN gets disconnected from internet, until the host machine is rebooted. The host machine has a little complicated network setup: the two gigabit connections are bonded and bridged to the VMs; however this set up has been serving me so well for several years now. The problem, on the other hand, appeared a few weeks ago. This doesn't happen every day but often enough to be annoying and disruptive for my family. My question is, how can I troubleshoot this problem and figure out whether it is truly due to network bridging somehow collapsing or not? I tried to find some log files but all I could find were the /var/log/libvirt/qemu/$VM files, and the particular log file for the pi-hole VM reported the following lines; however, I am not sure if they are associated with a real crash or just due to shutting down and restarting the host (please excuse the word-wrapping): char device redirected to /dev/pts/2 (label charserial0) qxl_send_events: spice-server bug: guest stopped, ignoring 2022-01-20T23:41:17.012445Z qemu-system-x86_64: terminating on signal 15 from pid 1 (/sbin/init) 2022-01-20 23:41:17.716+0000: shutting down, reason=crashed 2022-01-20 23:42:46.059+0000: starting up libvirt version: 7.10.0, qemu version: 6.2.0, kernel: 5.10.89-1-MANJARO, hostname: -redacted- Please excuse my ignorance but is there a way to restart the networking without rebooting the host machine? This will not solve my problem since I won't be able to reach to the host remotely if the networking is down. The real solution would be preventing these network crashes and the first step in that would be effective troubleshooting in my opinion. Any input/guidance will be greatly appreciated. I can provide more info about my host/VM(s) if the above is not adequate. Thanks, Hakan Duran

4 6

Backend memory object creation - query
by M, Shivakumar 27 Jan '22

27 Jan '22

Hello, For our use-case with Libvirt we want to create the Memory backend object , Expected QEMU args would be -object memory-backend-memfd,id=mem1,size=4096M Request you to please help us to specify this arg in the libvirt XML. Thanks, Shiv

2 1

cpuset.mems - Device or resource busy
by lejeczek 26 Jan '22

26 Jan '22

Hi guys On Centos 9 I get these in journal: ... libvirt version: 8.0.0, package: 1.el9 (builder(a)centos.org, 2022-01-14-15:00:06, ) hostname: dzien.mine.private Unable to write to '/sys/fs/cgroup/machine.slice/machine-qemu\x2d4\x2dubusrv2.scope/libvirt/cpuset.mems': Device or resource busy ... each time I do: -> virt-admin daemon-log-outputs There is more VMs running on the host but it's only these Ubuntu VMs Libvirt is not happy about. Any and all suggestions very appreciated. many thanks, L

1 0

Best practices for retrying QEMU P2P migrations
by Raphael Norwitz 24 Jan '22

24 Jan '22

Heya, We’ve recently hit a number of failures on QEMU P2P live migrations which appear to be caused by transient networking disconnects at different points in the migration process. We would like to implement smarter retry logic in our control plane to ensure such issues don’t stall critical workflows. On the other hand, we cannot blindly retry every failed migration because doing so greatly lengthens the time to fail high level autmation when there is a real problem. Are there currently any generally understood best practices for retrying migrations from a control plane perspective? Ideally we would decide whether or not to retry based on error codes, but especially in the QEMU P2P migration path many generic codes are returned. For example, see [1] where we attempted to improve an error code for a likely retry-able set of failure cases. [1] https://listman.redhat.com/archives/libvir-list/2022-January/msg00217.html Thanks, Raphael

1 0

What does the positional parameters of "virsh backup-begin" actually do?
by Ahmad Ismail 15 Jan '22

15 Jan '22

Normally when I backup a kvm machine I shutdown the machine then run: virsh list -all virsh shutdown Ubuntu18 virsh dumpxml Ubuntu18 > /MyBackup/Ubuntu18.xml cp /var/lib/libvirt/images/Ubuntu18.qcow2 /MyBackup/Ubuntu18.qcow2 However, I found a new sub-command. The help file says: % virsh backup-begin --help NAME backup-begin - Start a disk backup of a live domain SYNOPSIS backup-begin <domain> [--backupxml <string>] [--checkpointxml <string>] [--reuse-external] DESCRIPTION Use XML to start a full or incremental disk backup of a live domain, optionally creating a checkpoint OPTIONS [--domain] <string> domain name, id or uuid --backupxml <string> domain backup XML --checkpointxml <string> domain checkpoint XML --reuse-external reuse files provided by caller The problem with this help is, it is not clear enough. I understand that I should use virsh backup-begin vm1 to backup a live kvm machine. However, this command only create .qcow2 files. What about the .xml file. What does --backupxml , --checkpointxml & --reuse-external actually do? When should I use them?

2 2

interface hotplug q35 machine type
by Miguel Duarte de Mora Barroso 13 Jan '22

13 Jan '22

Hello, I see that as part of libvirt's documentation [0] the q35 machine type will feature at most 1 hotplugged PCIe device - default - and users must prepare in advance according to their expectations of how many ifaces will be hotplugged: """ Slots on the pcie-root controller do not support hotplug, so the device will be hotplugged into the pcie-root-port controller. If you plan to hotplug more than a single PCI Express device, you should add a suitable number of pcie-root-port controllers when defining the guest: for example, add ```xml <controller type='pci' model='pcie-root'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> ``` if you expect to hotplug up to three PCI Express devices, either emulated or assigned from the host. """ Is there any alternative to this ? For our use case, I'm considering mimicking Openstack's implementation - [1] - and expose a knob that indicates what is the number of PCIe root ports to be used upon the domain definition. I wonder how open would the community be to having a machine type alias that would provide a "better" default - in the sense that it would have more root port controllers. [0] - https://libvirt.org/pci-hotplug.html#x86_64-q35 [1] - https://blueprints.launchpad.net/nova/+spec/configure-amount-of-pcie-ports

2 1

DMARC & similar @admins
by lejeczek 12 Jan '22

12 Jan '22

Hi guys My memory fools me? - I remembered this list had DMAR+other_bits sorted, so users like myself, on Yahoo, did not loose their own emails. regards, L

2 3

'migrate' says it worked but in reality it did not - centOS 9
by lejeczek 12 Jan '22

12 Jan '22

Hi guys. I have a peculiar situation where between boxes: C->A -> $ virsh migrate --unsafe --live c8kubermaster1 qemu+ssh://10.1.1.99/system -> $ echo $? 0 but above does _not_ happen, instead!! VM was stopped in started, but _not_ migrated LIVE A->C -> $ virsh migrate --unsafe --live c8kubermaster1 qemu+ssh://10.1.1.100/system -> $ echo $? 0 indeed VM migrates live. box A & C have virtually identical OS stack, HW difference is: C = Ryzen 5 5600G A = Ryzen 5 3600 domain XML snippet where I think it matters: ... </metadata> <memory unit='GiB'>4</memory> <currentMemory unit='GiB'>4</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.6.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC-IBPB</model> <feature policy='require' name='ibpb'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='virt-ssbd'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> ... Initially I submitted a BZ against 'PCS' but continued to filled with it and I find 'libvirt' might be the culprit(also?) here. There is not much in logs, certainly nothing (with default verbosity) in virtqemud.service Is it that VM gets migrated but then is restarted on 'migrate_to' host? if so then why? How to start troubleshooting such 'monstrosity'? - all suggestions appreciated. many thanks, L.

2 4