October 2020 - Users - libvirt List Archives

[libvirt-users] [virtual interface] detach interface during boot succeed with no changes
by Yalan Zhang 08 Sep '22

08 Sep '22

Hi guys, when I detach an interface from vm during boot (vm boot not finished), it always fail. I'm not sure if there is an existing bug. I have confirmed with someone that for disk, there is similar behavior, if this is also acceptable? # virsh destroy rhel7.2; virsh start rhel7.2 ;sleep 2; virsh detach-interface rhel7.2 network 52:54:00:98:c4:a0; sleep 2; virsh dumpxml rhel7.2 |grep /interface -B9 Domain rhel7.2 destroyed Domain rhel7.2 started Interface detached successfully <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:98:c4:a0'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='rtl8139'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> When I detach after the vm boot, expand the sleep time to 10, it will succeed. # virsh destroy rhel7.2; virsh start rhel7.2 ;sleep 10; virsh detach-interface rhel7.2 network 52:54:00:98:c4:a0; sleep 2; virsh dumpxml rhel7.2 |grep /interface -B9 Domain rhel7.2 destroyed Domain rhel7.2 started Interface detached successfully ------- Best Regards, Yalan Zhang IRC: yalzhang Internal phone: 8389413

3 4

Problem with xen hvm DomU
by Christoph 08 Jan '21

08 Jan '21

Hi can someone say me whats wrong in this config: <domain type='xen'> <name>fenrir.chao5.int</name> <uuid>7aedcd03-54e8-4055-8d1b-37dd34194859</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='xenfv'>hvm</type> <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader> <kernel>/usr/lib64/xen/boot/hvmloader</kernel> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='variable' adjustment='0' basis='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/vg_astarte/lv_fenrir_root'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/vg_astarte/lv_fenrir_swap'/> <target dev='hdb' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='xenbus' index='0'/> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='00:16:3e:05:30:04'/> <source bridge='xenbr5'/> <script path='vif-bridge'/> <model type='netfront'/> </interface> <interface type='bridge'> <mac address='00:16:3e:04:30:04'/> <source bridge='xenbr4'/> <script path='vif-bridge'/> <model type='netfront'/> </interface> <interface type='bridge'> <mac address='00:16:3e:15:30:04'/> <source bridge='xenbr15'/> <script path='vif-bridge'/> <model type='netfront'/> </interface> <interface type='bridge'> <mac address='00:16:3e:14:30:04'/> <source bridge='xenbr14'/> <script path='vif-bridge'/> <model type='netfront'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='6000' autoport='no' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='8192' heads='1' primary='yes'/> </video> <memballoon model='xen'/> </devices> </domain> If I try to start it with virsh create... then I see only: error: Failed to create domain from /etc/libvirt/libxl/fenrir.chao5.int.xml error: internal error: libxenlight failed to create new domain 'fenrir.chao5.int' ------ Greetz

2 6

about the script /etc/qemu-ifup with nmcli command
by Yalan Zhang 06 Jan '21

06 Jan '21

Hi, I have tried the qemu-ifup script as below with nmcli command as brctl is deprecated on rhel8, but the guest network can not work. I think the script needs update. Could you please help to have a look? Thank you in advance. 1. prepare a linux bridge on the host named br0; 2. prepare the qemu-ifup script as below: # cat /etc/qemu-ifup #!/bin/bash # A br0 bridge should be already set up. # Compare with: # http://en.wikibooks.org/wiki/QEMU/Networking#qemu-ifup # # For the bridge setup, see: # http://wiki.libvirt.org/page/Networking#Fedora.2FRHEL_Bridging # http://gist.github.com/393525 ip link set "$1" up nmcli c add type bridge-slave ifname $1 con-name $1 master br0 autoconnect yes 3. start vm with below interface setting: # virsh dumpxml rh | grep /interface -B5 <interface type='ethernet'> <mac address='52:54:00:79:ba:dd'/> <script path='/etc/qemu-ifup'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> # virsh start rh Domain rh started 4.check on guest, the interface can not get dhcp ip address; 5. check on host, # nmcli con NAME UUID TYPE DEVICE br0 f68f73c7-10ee-40c1-bb09-3366d11ac896 bridge br0 ... vnet0 90a48d77-dccc-4b59-98f5-09f8cbd62458 ethernet -- # nmcli dev DEVICE TYPE STATE CONNECTION br0 bridge connected br0 ... vnet0 tun unmanaged -- 6. hotplug a bridge type interface and compare the tap devices: # virsh attach-interface rh bridge br0 --model virtio Interface attached successfully # nmcli con NAME UUID TYPE DEVICE br0 f68f73c7-10ee-40c1-bb09-3366d11ac896 bridge br0 vnet1 07c2a1f8-396f-4d5f-b61f-ef2ddb42ed93 tun vnet1 --->the hot-plugged one ... vnet0 90a48d77-dccc-4b59-98f5-09f8cbd62458 ethernet -- ----> the ethernet one # nmcli dev DEVICE TYPE STATE CONNECTION vnet1 tun connected (externally) vnet1 --->the hot-plugged one vnet0 tun unmanaged -- ----> the ethernet one ... 7. from the outputs above, the back-end tun device for ethernet type interface is unmanaged. I don't know how to update the script to fix it. Could you please help? ------- Best Regards, Yalan Zhang IRC: yalzhang

2 2

Isolated bridge does not bridge
by Paul van der Vlis 04 Jan '21

04 Jan '21

Hello, I want to do some testing and I have removed two VM's from the bridge what connects them to internet, and added them to another isolated bridge what's not connected to internet. Problem is that I cannot reach the other host in the isolated network. Something like this: virsh shutdown kvm66 virsh shutdown kvm68 brctl delif br0 vnet10 vnet6 # the interfaces of kvm66 and kvm68 brctl addbr br1 brctl addif br1 vnet10 vnet6 Then I've replaced br0 to br1 in the XML of both VM's with "virsh edit". Then I did start the VM's using the serial console (no network): virsh start --console kvm66 virsh start --console kvm68 I cannot ping from one machine to the other. Why?? With regards, Paul -- Paul van der Vlis Linux systeembeheer Groningen https://www.vandervlis.nl/

4 5

Set hostname of guest during installation time
by john doe 05 Nov '20

05 Nov '20

Hi, I would like to set the hostname when installing a guest, with the below command the hostname is not set to 'try06' in the guest: virt-install --name=try06 --graphic none --pxe --network bridge=virbr0 How can I set the hostname of the guest during installation time? I realy appriciate the support I'm getting in here, I'm fairly new to libvirt. -- John Doe

4 24

proper config for qemu's host_cdrom
by daggs 03 Nov '20

03 Nov '20

Greetings, I was wondering what is the proper way to configure a scsi cdrom pass-through so in when the qemu line is generated, host_Cdrom will be used instead of host_device. looking at https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_block.c#L1090, I see that hostcdrom must be true. in order for that to be true, the following must be (see https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_domain.c#L71…) 1. disk->device == VIR_DOMAIN_DISK_DEVICE_CDROM 2. disksrc->format == VIR_STORAGE_FILE_RAW 3. virStorageSourceIsBlockLocal(disksrc) 4. virFileIsCDROM(disksrc->path) == 1 virFileIsCDROM uses the kernel, so I assume that as disksrc->path points to the actual path (I can see it in the qemu line) than #4 returns 1. the other 3 are more complicated. my xml snippet is this: <devices> <hostdev mode='subsystem' type='scsi' rawio='yes'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> </devices> Thanks, Dagg

2 2

about the new added attributes "check" and "type" for interface mac element
by Yalan Zhang 02 Nov '20

02 Nov '20

Hi all, I have done some tests for the new attributes "check" and "type", could you please help to have a check? And I have some questions about the patch, please help to have a look, Thank you! The questions: 1. in step 4 below, the error message should be updated: Actual results: XML error: invalid mac address **check** value: 'next'. Valid values are "generated" and "static". expected results: XML error: invalid mac address **type** value: 'next'. Valid values are "generated" and "static". 2. I have checked the vmware OUI definition and found this: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking… it says the VMware OUI is 00:50:56, not 00:0c:29 in the patches. Am I missing something? 3. Could you please tell more about the user story? as I can not understand the scenario when " it will ignore all the checks libvirt does about the origin of the MAC address(whether or not it's in a VMWare OUI) and forward the original one to the ESX server telling it not to check it either". Does it happen when we try to transform a kvm guest to a vmware guest? 4. How to test it as a libvirt QE? Are the test scenarios below enough without ESX env? Test steps: 1. Start vm with different configuration with mac in "00:0c:29" range: # virsh dumpxml rhel | grep /interface -B12 ... <interface type='network'> <mac address='00:0c:29:e7:9b:cb' type='generated' check='yes'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> <interface type='network'> <mac address='00:0c:29:3b:e0:50' type='static' check='no'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/> </interface> <interface type='network'> <mac address='00:0c:29:73:f6:dc' type='generated' check='no'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/> </interface> <interface type='network'> <mac address='00:0c:29:aa:dc:6c' type='static' check='yes'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/> </interface> # virsh start rhel Domain rhel started 2. login guest and check the interfaces: # ip addr ... 2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:e7:9b:cb brd ff:ff:ff:ff:ff:ff inet 192.168.122.142/24 brd 192.168.122.255 scope global dynamic noprefixroute enp5s0 valid_lft 3584sec preferred_lft 3584sec inet6 fe80::351c:686a:863e:4a7f/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: enp11s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:3b:e0:50 brd ff:ff:ff:ff:ff:ff inet 192.168.122.202/24 brd 192.168.122.255 scope global dynamic noprefixroute enp11s0 valid_lft 3584sec preferred_lft 3584sec inet6 fe80::2b79:4675:6c59:6822/64 scope link noprefixroute valid_lft forever preferred_lft forever 4: enp12s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:73:f6:dc brd ff:ff:ff:ff:ff:ff inet 192.168.122.33/24 brd 192.168.122.255 scope global dynamic noprefixroute enp12s0 valid_lft 3584sec preferred_lft 3584sec inet6 fe80::e43d:555:ba85:4030/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: enp13s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:0c:29:aa:dc:6c brd ff:ff:ff:ff:ff:ff inet 192.168.122.161/24 brd 192.168.122.255 scope global dynamic noprefixroute enp13s0 valid_lft 3584sec preferred_lft 3584sec inet6 fe80::f32d:e2e8:9c8b:47fd/64 scope link noprefixroute valid_lft forever preferred_lft forever 3. start vm without the "check" and "type" attributes, and check the live xml do not include these attributes, either. # virsh start vm1 virshDomain vm1 started # virsh dumpxml vm1 | grep /interface -B8 </controller> <interface type='network'> <mac address='52:54:00:bb:cd:89'/> <source network='default' portid='b02dc78f-69ad-4db7-870c-f371fd730537' bridge='virbr0'/> <target dev='vnet22'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> 4. negative test: Set "<mac address='52:54:00:bb:cd:89' type='next'/>" in virsh edit # virsh edit vm1 error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content Failed. Try again? [y,n,i,f,?]: ----> press 'i' error: XML error: invalid mac address check value: 'next'. Valid values are "generated" and "static". Failed. Try again? [y,n,f,?]: ------- Best Regards, Yalan Zhang IRC: yalzhang

1 1

virsh rights voor normal users
by Natxo Asenjo 30 Oct '20

30 Oct '20

hi, using the cockpit web ui and with these instructions: https://libvirt.org/dbus.html#usage we allow successfully that a group of users can access the console of the system vms in different kvm hosts. Oddly enough, in the same cockpit web interface I can use a terminal, and if I run virsh list --all I get an empty listing. So using cockpit I can manage the system vms, but I cannot use virsh. This is in a rhel 7.8 system. The host is joined to an Idm realm, and this realm has a trust to an AD forest. The users are AD users mapped to an external Idm group. Any ideas as to what we do wrong? Thanks! -- regards, Natxo

4 6

Fail to build libvirt upstream rpm on fedora 33
by Han Han 30 Oct '20

30 Oct '20

Hello, I encountered errors when building rpm on fedora33: Versions: libvirt: v6.9.0-rc2 gcc-10.2.1-5.fc33.x86_64 meson-0.55.3-1.fc33.noarch ninja-build-1.10.1-2.fc33.x86_64 binutils-2.35-11.fc33.x86_64 Steps: 1. Archive libvirt source code to ~/rpmbuild/SOURCES/libvirt-6.9.0.tar.xz 2. Copy libvirt.spec.in to ~/rpmbuild/SPECS/libvirt.spec. Then `sed -i s/@VERSION@/6.9.0/g ~/rpmbuild/SPECS/libvirt.spec` 3. Build rpm: $ rpmbuild -ba ~/rpmbuild/SPECS/libvirt.spec Then errors happen: [1071/1255] gcc -o tests/qemucapsprobe tests/qemucapsprobe.p/qemucapsprobe.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,--start-group tests/libtest_qemu_driver.so src/libvirt.so.0.6009.0 -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/ libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,--end-group '-Wl,-rpath,$ORIGIN/:$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src FAILED: tests/qemucapsprobe gcc -o tests/qemucapsprobe tests/qemucapsprobe.p/qemucapsprobe.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,--start-group tests/libtest_qemu_driver.so src/libvirt.so.0.6009.0 -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/ libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,--end-group '-Wl,-rpath,$ORIGIN/:$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src /usr/bin/ld: tests/libtest_qemu_driver.so: undefined reference to `__open_missing_mode' collect2: error: ld returned 1 exit status [1072/1255] gcc -o tests/virtimetest src/libvirt_probes.o tests/virtimetest.p/virtimetest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/ libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src [1073/1255] gcc -o tests/virstringtest src/libvirt_probes.o tests/virstringtest.p/virstringtest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/ libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src [1074/1255] gcc -o tests/virsystemdtest src/libvirt_probes.o tests/virsystemdtest.p/virsystemdtest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/ libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src [1075/1255] gcc -o tests/virtypedparamtest src/libvirt_probes.o tests/virtypedparamtest.p/virtypedparamtest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/ libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src [1076/1255] gcc -o tests/domaincapstest src/libvirt_probes.o tests/domaincapstest.p/domaincapstest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a tests/libtest_file_wrapper.a tests/libtest_utils_xen.a tests/libtest_utils_qemu.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 tests/libtest_xen_driver.so tests/libtest_qemu_driver.so -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/ libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src:$ORIGIN/' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests FAILED: tests/domaincapstest gcc -o tests/domaincapstest src/libvirt_probes.o tests/domaincapstest.p/domaincapstest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a tests/libtest_file_wrapper.a tests/libtest_utils_xen.a tests/libtest_utils_qemu.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 tests/libtest_xen_driver.so tests/libtest_qemu_driver.so -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/ libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src:$ORIGIN/' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests /usr/bin/ld: tests/libtest_qemu_driver.so: undefined reference to `__open_missing_mode' collect2: error: ld returned 1 exit status [1077/1255] gcc -o tests/vircapstest src/qemu/libvirt_qemu_probes.o src/libvirt_probes.o tests/vircapstest.p/vircapstest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a tests/libtest_utils_lxc.a tests/libtest_utils_qemu.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 src/lxc/libvirt_driver_lxc_impl.a src/qemu/libvirt_driver_qemu_impl.a -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/ libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so /usr/lib64/libblkid.so -lcap-ng /usr/lib64/libfuse.so -pthread -lselinux -Wl,-z,relro -Wl,-z,now -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -lcap-ng -lselinux -Wl,-z,relro -Wl,-z,now -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src [1078/1255] gcc -o tests/cputest src/libvirt_probes.o tests/cputest.p/cputest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a tests/libtest_utils_qemu.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 tests/libtest_utils_qemu_monitor.a tests/libtest_qemu_driver.so -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/ libglib-2.0.so /usr/lib64/libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src:$ORIGIN/' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests FAILED: tests/cputest gcc -o tests/cputest src/libvirt_probes.o tests/cputest.p/cputest.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-export-dynamic -pie -Wl,--whole-archive -Wl,--start-group tests/libtest_utils.a tests/libtest_utils_qemu.a -Wl,--no-whole-archive -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld src/libvirt.so.0.6009.0 tests/libtest_utils_qemu_monitor.a tests/libtest_qemu_driver.so -Wl,--no-copy-dt-needed-entries -Wl,-export-dynamic -ldl /usr/lib64/libglib-2.0.so /usr/lib64/ libgobject-2.0.so /usr/lib64/libgio-2.0.so /usr/lib64/libgnutls.so /usr/lib64/libnl-3.so /usr/lib64/libnl-route-3.so /usr/lib64/libxml2.so /usr/lib64/libsasl2.so -lselinux /usr/lib64/libtirpc.so /usr/lib64/libyajl.so -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,-export-dynamic -lselinux -Wl,--end-group '-Wl,-rpath,$ORIGIN/../src:$ORIGIN/' -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/src -Wl,-rpath-link,/home/hhan/rpmbuild/BUILD/libvirt-6.9.0/x86_64-redhat-linux-gnu/tests /usr/bin/ld: tests/libtest_qemu_driver.so: undefined reference to `__open_missing_mode' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. error: Bad exit status from /var/tmp/rpm-tmp.xEqkEF (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.xEqkEF (%build)

2 2

virt-install with pool off glusterfs
by lejeczek 28 Oct '20

28 Oct '20

Hi Guys, what can, is, wrong with that syntax of mine below? $ virt-install -n fcoskubermaster1 --virt-type kvm --os-variant fedora31 --graphics="none" --memory $((4*1024)) --disk="pool=gluster-VMs,size=10,path=/fcoskubermaster1.qcow2,backing_store=gluster://127.0.0.1/VMs/00-BACKing/fcoskubermaster1.qcow2" --network network=ovsbr0,model=virtio --accelerate --cpuset 3 --cpu host-model --vcpus 3 --import --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=/00.GLUSTER-mnt/VMs/fcos.ign" WARNING fcoskubermaster1 may not be accessible by the hypervisor. You will need to grant the 'qemu' user search permissions for the following directories: [''] Starting install... ERROR Couldn't create storage volume 'fcoskubermaster1': 'this function is not supported by the connection driver: storage pool does not support volume creation' Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf start fcoskubermaster1 otherwise, please restart your installation. Basically what I hope to do is to have backing image and the actual one on a glusterfs volume, backing store in /00-BACKing and the image in / off that same gluster-VMs volume. many thanks, L.

1 0

RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)
by Marcel Juffermans 28 Oct '20

28 Oct '20

Hi there, Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the virtual guests don't get their RBD disks made available to them. On openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML is as follows: <domain type='xen' id='7'> <name>mytwotel-a</name> <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid> <description>mytwotel-a</description> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8-23'/> </cputune> <os> <type arch='x86_64' machine='xenpv'>linux</type> <kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writethrough'/> <source protocol='rbd' name='guests/mytwotel-a'> <auth username='libvirt'> <secret type='ceph' uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/> </auth> </source> <backingStore/> <target dev='xvda' bus='xen'/> </disk> <controller type='xenbus' index='0'/> <interface type='bridge'> <mac address='00:16:3e:a3:ba:9f'/> <source bridge='br0'/> <target dev='vif7.0'/> </interface> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='xen' port='0'/> </console> <input type='mouse' bus='xen'/> <input type='keyboard' bus='xen'/> <memballoon model='xen'/> </devices> </domain> The virtual guest starts, but then sits in the Grub 2 boot prompt because the disk is not available. The qemu log shows: qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory qemu-system-i386: failed to create 'qdisk' device '51712': failed to create drive: Could not open 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': No such file or directory ... I tried to strace libvirtd. The results are as follows: On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this: 1682 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory) 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000 1682 mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM IO], 8) = 0 1682 rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0 1682 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...> 1682 <... mmap resumed>) = 0x7f538adfc000 1682 mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...> 1682 <... mprotect resumed>) = 0 1682 rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...> 1682 <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0 1682 write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...> ... On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see this: 16267 openat(AT_FDCWD, "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory) 16267 stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory) 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", O_RDONLY|O_CLOEXEC) = 60 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 &\0\0\0\0\0\0"..., 832) = 832 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000 16267 close(60) = 0 ... Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library *does* exist on openSUSE 15.2 but it doesn't seem to be used. I've tried to update libvirt to a newer version using the Open Build Service repos, but then ran into so many conflicting versions that I gave up. At this point I'm stuck. Does anyone have an idea I can try? Many thanks, Marcel

3 5

Understanding 'State change failed: (-2) Need access to UpToDate data'
by Paul O'Rorke 28 Oct '20

28 Oct '20

Hi list, I had to relocate my third node in a classic DRBD 8.4 three node set up to a new host. I am having difficulty making the stacked resource the primary. I am following this guide: https://www.linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-three-nodes Specifically this: > > > 5.18.3. Enabling stacked resources > > To enable a stacked resource, you first enable its lower-level > resource and promote it: > > drbdadm up r0 > drbdadm primary r0 > > As with unstacked resources, you must create DRBD meta data on the > stacked resources. This is done using the following command: > > # drbdadm create-md --stacked r0-U > > Then, you may enable the stacked resource: > > # drbdadm up --stacked r0-U > # drbdadm primary --stacked r0-U > > After this, you may bring up the resource on the backup node, enabling > three-node replication: > > # drbdadm create-md r0-U > # drbdadm up r0-U It seems all is good right up to the last command: :~# drbdadm primary informer :~# drbdadm create-md --stacked informer-U You want me to create a v08 style flexible-size internal meta data block. There appears to be a v08 flexible-size internal meta data block already in place on /dev/drbd11 at byte offset 429483581440 Do you really want to overwrite the existing meta-data? [need to type 'yes' to confirm] yes md_offset 429483581440 al_offset 429483548672 bm_offset 429470441472 Found some data ==> This might destroy existing data! <== Do you want to proceed? [need to type 'yes' to confirm] yes initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created. :~# drbdadm up --stacked informer-U :~# drbdadm primary --stacked informer-U 110: State change failed: (-2) Need access to UpToDate data Command 'drbdsetup-84 primary 110' terminated with exit code 17 drbd11 is the device for the lower level resource 'informer', drbd110 the device for stacked resource 'informer-U'. Replication between hosts 'Alice' and 'Bob' (not the real host names but corresponding to the names used int the documentation) is functional. I am performing the above commands on 'bob'. I do not understand why it is complaining it needs UpToDate data in that last command if 'informer-U' is 'stacked-on-top-of informer' and was just created. the .res file(s) look like this: > resource informer { > net { > protocol C; > } > > device /dev/drbd11; > meta-disk internal; > > on trk-kvm-01 { > address 10.10.1.125:7789; > disk /dev/trk-kvm-01-vg/informer; > } > on trk-kvm-02 { > address 10.10.1.126:7789; > disk /dev/trk-kvm-02-vg/informer; > } > } > > resource informer-U { > net { > protocol A; > } > > stacked-on-top-of informer { > device /dev/drbd110; > address 10.10.2.126:7789; > } > > on trk-bkp-01 { > device /dev/drbd110; > disk /dev/ubuntu-vg/informer; > address 10.10.2.127:7789; > meta-disk internal; > } > } Any suggestions on what I am missing in this picture? Please and thanks. -- Paul O'Rorke Tracker Software Products (Canada) Limited www.tracker-software.com <https://www.tracker-software.com> Tel: +1 (250) 324 1621 Fax: +1 (250) 324 1623 Support: www.tracker-software.com/support <https://www.tracker-software.com/support> Download latest Releases: www.tracker-software.com/downloads <https://www.tracker-software.com/downloads>

1 0

Why "discard":"unmap" is the default option for disks
by Han Han 27 Oct '20

27 Oct '20

Hello, I find "discard":"unmap" is defaultly enabled in qemu cmdline(libvirt v6.6, qemu v5.1): XML: <disk type="file" device="disk"> <driver name="qemu" type="qcow2"/> <source file="/var/lib/libvirt/images/new.qcow2" index="2"/> <backingStore/> <target dev="sda" bus="scsi"/> <alias name="scsi0-0-0-0"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> </disk><disk type="network" device="disk"> <driver name="qemu" type="raw" error_policy="report"/> <source protocol="nbd" name="new" tls="no" index="1"> <host name="localhost" port="10809"/> </source> <target dev="vdb" bus="virtio"/> <alias name="virtio-disk1"/> <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/> </disk> QEMU cmdline: ... -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/new.qcow2","node-name":"libvirt-2-storage","auto-read-only":true, *"discard":"unmap"*} -blockdev {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-2-format,id=scsi0-0-0-0,bootindex=1 -blockdev {"driver":"nbd","server":{"type":"inet","host":"localhost","port":"10809"},"export":"new","node-name":"libvirt-1-storage","auto-read-only":true, *"discard":"unmap"*} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"} -device virtio-blk-pci,bus=pci.2,addr=0x0,drive=libvirt-1-format,id=virtio-disk1,werror=report,rerror=report ... I think it's from https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_block.c#L1211 But I cannot find the reason from commit msgs or documents. Could you please explain it?

2 2

unable to migrate: virPortAllocatorSetUsed:299 : internal error: Failed to reserve port 49153
by Vjaceslavs Klimovs 27 Oct '20

27 Oct '20

On libvirt 6.8.0 and qemu 5.1.0, when trying to live migrate "error: internal error: Failed to reserve port" error is received and migration does not succeed: virsh # migrate cartridge qemu+tls://ratchet.lan/system --live --persistent --undefinesource --copy-storage-all --verbose error: internal error: Failed to reserve port 49153 virsh # On target host with debug logs, nothing interesting but the error itself is found in the logs ... 2020-10-12 02:11:33.852+0000: 6871: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"return": {}, "id": "libvirt-373"}] 2020-10-12 02:11:33.852+0000: 6871: info : qemuMonitorJSONIOProcessLine:239 : QEMU_MONITOR_RECV_REPLY: mon=0x7fe784255020 reply={"return": {}, "id": "libvirt-373"} 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjExitMonitorInternal:5615 : Exited monitor (mon=0x7fe784255020 vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjEndJob:1140 : Stopping job: async nested (async=migration in vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: error : virPortAllocatorSetUsed:299 : internal error: Failed to reserve port 49153 2020-10-12 02:11:33.852+0000: 6825: debug : qemuMigrationParamsReset:1206 : Resetting migration parameters 0x7fe784257c30, flags 0x59 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjBeginJobInternal:835 : Starting job: job=async nested agentJob=none asyncJob=none (vm=0x55f086c81ea0 name=cartridge, current job=none agentJob=none async=migration in) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjBeginJobInternal:887 : Started job: async nested (async=migration in vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuDomainObjEnterMonitorInternal:5590 : Entering monitor (mon=0x7fe784255020 vm=0x55f086c81ea0 name=cartridge) 2020-10-12 02:11:33.852+0000: 6825: debug : qemuMonitorSetMigrationCapabilities:3853 : mon:0x7fe784255020 vm:0x55f086c81ea0 fd:30 2020-10-12 02:11:33.852+0000: 6825: info : qemuMonitorSend:944 : QEMU_MONITOR_SEND_MSG: mon=0x7fe784255020 msg={"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability> fd=-1 ... Full logs: destination: https://drive.google.com/file/d/1g986SbSVijvwZd8d7xDrJwo_AmKH-JnV/view?usp=… source: https://drive.google.com/file/d/1lsV2EOBxF7xH5-lgz2Psh9YSkOePXvAd/view?usp=… On target host, nothing is listening on the target port: ratchet /var/log/libvirt/qemu # netstat -lnp | grep 49153 | wc -l 0 and nc -l 49153 succeeds without issues. Do you have any suggestions on how to proceed here?

2 4

possible bug in efi detection for guest
by daggs 20 Oct '20

20 Oct '20

Greetings All, following a suggestion I got here on how to properly boot uefi-q35 guest, I found an weird config in the xml. this is what I see when I run virsh edit streamer-vm-q35: <os firmware='efi'> <type arch='x86_64' machine='pc-q35-5.0'>hvm</type> <boot dev='hd'/> </os> when I run virsh dumpxml streamer-vm-q35, I get this: <os> <type arch='x86_64' machine='pc-q35-5.0'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/edk2-x86_64-secure-code.fd</loader> <nvram template='/usr/share/qemu/edk2-i386-vars.fd'>/var/lib/libvirt/qemu/nvram/streamer-vm-q35_VARS.fd</nvram> <boot dev='hd'/> </os> question is, why the nvram template is /usr/share/qemu/edk2-i386-vars.fd and not /usr/share/qemu/edk2-x86_64-code.fd (file exists) as the system is x64? Thanks Dagg.

2 2

scsi passthrough differs between guests
by daggs 15 Oct '20

15 Oct '20

Greetings, I have two machines running the same distro, both running qemu 5.1.0, one runs libvirt 6.7.0, the other 6.8.0. I've decided to test the viability of passing through my sata cdrom into a vm, so I went to the libvirt docs, read a bit and added the following to a debian10 uefi vm running on libvirt 6.8.0: <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </controller> <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host11'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> I've booted the vm and saw the cdrom in the vm, eject worked and so did mount. I've decided to move it to another machine running libvirt 6.7.0, I've verified the cdrom is visible on the new system, seE: # lsscsi [0:0:0:0] cd/dvd HL-DT-ST DVDRAM GH24NSD1 LW00 /dev/sr0 I've inserted the same xml into another vm running libreelec uefi but changed the host from 11 to 0, it looks like this: <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <readonly/> <alias name='hostdev0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> but when I boot the vm, I see no cdrom, lspci shows this: 03:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) 07:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI as there is no lsscsi or /proc/scsi/scsi I cannot see other possible scsis. I know for a fact that libreelec supports scsi/sata cdroms as it is a frequent he for streamers. I'm baffled about the two scsi controllers I see and why I cannot see the device. is there a known issue in 6.7.0 with scsi pass-trough? Dagg.

2 6

libivrt client using python on windows
by Talha Jawaid 14 Oct '20

14 Oct '20

Hello, I want to run a python script on Windows to remotely control libvirt running on a (Linux) server. This was all working fine while prototyping stuff on the server but now I am having some trouble installing the python module on windows ("pip install" fails). I struggled through getting it to COMPILE but then ran into linking issues. Seems like the actual libvirt library is needed to run (and even compile) on Windows? Is there a prebuilt package somewhere that I could get? Do I really need the whole libvirt library or is there a client only package? How could I go about accomplishing my goal here? Thanks, -Talha

2 1

unable to find any master var store for loader error
by daggs 14 Oct '20

14 Oct '20

Greetings, I have the following machine: https://dpaste.com/5BPA3F77F which I'm trying to boot in uefi. /etc/libvirt/qemu.conf looks like this: https://dpaste.com/B3SFHUY6R and the ovmf files exists in the path, see: # ll /usr/share/edk2-ovmf/OVMF_CODE.fd /usr/share/edk2-ovmf/OVMF_VARS.fd /usr/share/edk2-ovmf/OVMF_CODE.secboot.fd /usr/share/edk2-ovmf/OVMF_VARS.secboot.fd -rw-r--r-- 1 root root 1966080 Aug 21 14:32 /usr/share/edk2-ovmf/OVMF_CODE.fd -rw-r--r-- 1 root root 1966080 Aug 21 14:32 /usr/share/edk2-ovmf/OVMF_CODE.secboot.fd -rw-r--r-- 1 root root 131072 Aug 21 14:32 /usr/share/edk2-ovmf/OVMF_VARS.fd -rw-r--r-- 1 root root 131072 Aug 21 14:32 /usr/share/edk2-ovmf/OVMF_VARS.secboot.fd when I try to start the machine, I get this error: error: Failed to start domain vm1 error: operation failed: unable to find any master var store for loader: /usr/share/edk2-ovmf/OVMF_CODE.fd libvirt version is 6.7.0 and qemu version is 5.1.0 any idea how to fix this issue? Thanks. Dagg.

3 6

Re: Encrypting boot partition Libvirt not showing the OS booting up
by john doe 13 Oct '20

13 Oct '20

On 10/12/2020 11:53 AM, john doe wrote: > On 10/12/2020 11:37 AM, Peter Krempa wrote: >> On Mon, Oct 12, 2020 at 11:27:20 +0200, john doe wrote: >>> Hi, thank you for your answer, I'm sending this privatly as you asked >>> for private information. >>> Can I ask you to keep those information private? >>> >>> On 10/12/2020 10:29 AM, Peter Krempa wrote: >>>> On Mon, Oct 12, 2020 at 10:03:15 +0200, john doe wrote: >>>>> Hi, >>>>> >>>>> I have installed Debian Buster with encrypted LVM so apon installation >>>>> my root partition is encrypted. >>>>> So far so good but as soon as I encrypt the boot partition, after >>>>> reboot >>>>> the OS won't start. >>>>> If I start the drive directly with qemu, it works but it looks like >>>>> Libvirt is somehow not able to deel with it. >>>> >>>> This is not enough information to diagnose the problem. >>>> >>>> We'll need the following: >>>> >>>> 1) Did you encrypt the partition using the debian installer >>>> >>> >>> No, I did it after installation following the instructions at (1). >>> >>>> 2) what vm XML you used: >>>> a) during installation >>> >>> The domain xml file was created by virt-install with the following >>> command: >>> $ virsh destroy try01; virsh undefine try01; time virt-install >>> --name=try01 --ram=1024 --noreboot --cpuset=auto --cpu host >>> --vcpus=1,maxvcpus=4 --disk=path=/mnt/usbkey01/machines/try/try01,size=6 >>> --graphic none --pxe --os-variant=debian10 --network >>> bridge=br0,mac=0e:35:32:84:c3:f3 --filesystem >>> type=mount,mode=mapped,source=/mnt/usbkey01/public,target=public_dir >>> >>>> b) when trying to finally boot the vm >>>> >>> >>> Attached as 'try01.xml' obtained by doing 'virsh dumpxml try01 > >>> try01.xml'. >>> >>>> 3) what qemu command line you've used at the point you claim it worked >>>> >>> >>> qemu-system-x86_64 -drive file=/mnt/usbkey01/machines/try/try01 -m 1024 >>> -boot c -accel kvm -machine q35 -nographic >>> >>>> 4) what is the error/final state when the VM fails to boot with libvirt >>>> >>> >>> After having encrypted the boot partition: >>> >>> $virsh console try01 >>> root@0e-35-32-84-c3-f3:# [ 208.513259] watchdog: watchdog0: watchdog >>> did not stop! >>> [ 208.855971] reboot: Restarting system >>> >>> >>> $ qemu-system-x86_64 -drive file=/mnt/usbkey01/machines/try/try01 -m >>> 1024 -boot c -accel kvm -machine q35 -nographic >>> SeaBIOS (version 1.12.0-1) >>> >>> >>> iPXE (http://ipxe.org) 00:02.0 C980 PCI2.10 PnP PMM+3FF8FE80+3FECFE80 >>> C980 >>> >>> >>> >>> Booting from Hard Disk... >>> Attempting to decrypt master key... >>> Enter passphrase for hd0,msdos1 (43a322dfc8ba4628b80afc66d49642a7): >>> >>> >>> As you can see above, if I invoked qemu directly, I get prompted for the >>> boot passthrase but I'm not getting it when using libvirt. >> >> Okay, so the root cause ... or "problem" here is that you don't see the >> console via 'virsh console' after the guest OS rebooted. >> >> My suspicion according to the VM XML is that the VM restart triggered a >> restart of the qemu process and thus our internal handler of the console >> passthrough disconnected. >> >> Please try a "virsh destroy $VM" if the VM is running/stuck waiting for >> the password without actually showing it and then start it using >> >> virsh start --console $VM >> >> This will start it and connect to the console immediately. >> >> Please report your findings, we might want to discuss what happens when >> a console is connected and the guest uses a setting of: >> >> <on_reboot>restart</on_reboot> >> >> when the VM is rebooted. >> > > Still no luck: > > $ virsh destroy try01; virsh start --console try01 > error: Failed to destroy domain try01 > error: Requested operation is not valid: domain is not running > > Domain try01 started > Connected to domain try01 > Escape character is ^] > > > > > > I did not modify anything during the time I sent my answer and seeing > yours. > I sent privately the requested xml file to 'Peter Krempa <pkrempa(a)redhat.com>'. Peter Krempa 's privately answered me back suggesting to add the following in the domain xml file: <bios useserial='yes'/> under <os> such as ... <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <boot dev='hd'/> <bios useserial='yes'/> </os> This does not help at all and still give the output sent previously. -- John Doe

5 12

Re: Attached disk blockpull
by Peter Krempa 13 Oct '20

13 Oct '20

This is a libvirt question, so asking it on the qemu-block might not get you an answer that quick, ... or ever if I didn't notice your other question also addressed incorrectly. [adding libvirt-users(a)redhat.com to cc] On Tue, Sep 01, 2020 at 10:57:34 -0400, Yoonho Park wrote: > I am trying to perform a blockpull on an attached disk, but I cannot do > this using "virsh attach-disk" specifying a new, empty overlay file created > with "qemu-img create". The VM does not pick up the backing_file path from Could you please elaborate on the exact steps? specifically the arguments for qemu-img create. My feeling is that you didn't specify the backing image format (-F) when creating the overlay, but it's hard to tell from this very sparse question. > the qcow2 overlay file, and there seems to be no way to specify the backing > path through the attach-disk command itself. Instead the backing path has virsh attach-disk is a syntax-sugar wrapper which generates the XML document which is in fact used to attach disk. You can use virsh attach-disk --print-xml to see what would be used. > to be specified in the xml file used to create the VM, which precludes > attaching a new disk to a running VM. Is this a bug? Is it possible to using virsh attach-device allows you to pass in a whole <devices> subelement such as <disk> with any configuration you are able to specify when defining the XML. The virsh attach-disk is limited to just "basic" config options. > attach a disk to a running VM, and specify its backing path, using qemu > directly? This is with qemu 4.2.0 and libvirt 6.0.0.

1 0

Failed to get udev device for syspath '/sys/devices/virtual/dmi/id'
by Mathieu Tarral 12 Oct '20

12 Oct '20

Hi, I'm facing an issue with libvirt and the LIBXL driver, failing when searching for DMI data in /sys. info : libvirt version: 5.0.0, package: 4+deb10u1 (Guido Günther <agx(a)sigxcpu.org> Thu, 05 Dec 2019 00:22:14 +0100) error : udevGetDMIData:1719 : internal error: Failed to get udev device for syspath '/sys/devices/virtual/dmi/id' or '/sys/class/dmi/id' error : libxlDriverConfigNew:1803 : Unable to configure libxl's memory management parameters error : virStateInitialize:662 : Initialization of LIBXL state driver failed: internal error: Failed to get udev device for syspath '/sys/devices/virtual/dmi/id' or '/sys/class/dmi/id' error : daemonRunStateInit:799 : Driver state initialization failed The relevant function udevGetDMIData in libvirt: https://github.com/libvirt/libvirt/blob/cb09344a2cccc0cc9bcefa3cb53d7af45ba… I don't understand what's happening, as this initialization failure only happens when I boot on the Xen kernel. Also, the /sys/class/dmi* paths are not exposed on Xen. Note: Running on Xen 4.12.1 Any ideas ? Thanks !

4 5

Encrypting boot partition Libvirt not showing the OS booting up
by john doe 12 Oct '20

12 Oct '20

Hi, I have installed Debian Buster with encrypted LVM so apon installation my root partition is encrypted. So far so good but as soon as I encrypt the boot partition, after reboot the OS won't start. If I start the drive directly with qemu, it works but it looks like Libvirt is somehow not able to deel with it. What am I missing? -- John Doe

2 1

Building libvirt library without libvirtd or virsh
by Román González 09 Oct '20

09 Oct '20

Hello there, I'm trying to play with musl and libvirt to see if I'm able to build a libvirt client binary without dynamic lib dependencies. I have two questions: 1) to your knowledge, is this exercise futile? 2) Do you know if there is a way to *only* compile the library bits? I want to reduce the number of dependencies in the build, and only construct the libvirt libraries, not libvirtd nor virsh. Thanks in advance for any help you can provide. Roman.-

3 4

Is it possible that "virsh destroy" does not stop a domain ?
by Lentes, Bernd 08 Oct '20

08 Oct '20

Hi, Is it possible that "virsh destroy" does not stop a domain ? I'm asking because i have some domains running in a two-node HA-Cluster (pacemaker). And sometimes one node get fenced (killed) because it couldn't stop a domain. That's very ugly. This is also the reason why i asked before what "virsh destroy" really does ? IIRC a kill -9 can't terminate a process which is in "D" state (uninterruptible sleep). So if the process of the domain is in "D" state, it can't be finished. Right ? Pacemaker tries to shutdown or destroy a domain with a resource agent, which is a shell script, similar to an init script. Here is an excerp from the resource agent for virtual domains: force_stop() { local out ex translate local status=0 ocf_log info "Issuing forced shutdown (destroy) request for domain ${DOMAIN_NAME}." out=$(LANG=C virsh $VIRSH_OPTIONS destroy ${DOMAIN_NAME} 2>&1) # hier wird die domain destroyed ex=$? translate=$(echo $out|tr 'A-Z' 'a-z') echo >&2 "$translate" case $ex$translate in *"error:"*"domain is not running"*|*"error:"*"domain not found"*|\ *"error:"*"failed to get domain"*) : ;; # unexpected path to the intended outcome, all is well sucess [!0]*) ocf_exit_reason "forced stop failed" # <============ fail of destroy seems to be possible return $OCF_ERR_GENERIC ;; 0*) while [ $status != $OCF_NOT_RUNNING ]; do VirtualDomain_status status=$? done ;; esac return $OCF_SUCCESS } The function force_stop is responsible for stop/destroy the domain. And it cares about a non-working "virsh destroy". Is there a developer who can explain what "virsh destroy" really does ? Or is there another ML for the developers ? Bernd -- Bernd Lentes Systemadministration Institute for Metabolism and Cell Death (MCD) Building 25 - office 122 HelmholtzZentrum München bernd.lentes(a)helmholtz-muenchen.de phone: +49 89 3187 1241 phone: +49 89 3187 3827 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/mcd stay healthy Helmholtz Zentrum MÃ¼nchen Helmholtz Zentrum MÃ¼nchen

3 3

PCI Passthrough and Surprise Hotplug
by Marc Smith 08 Oct '20

08 Oct '20

Hi, I'm using QEMU/KVM on RHEL (CentOS) 7.8.2003: # cat /etc/redhat-release CentOS Linux release 7.8.2003 I'm passing an NVMe drive into a Linux KVM virtual machine (<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>) which has the following 'hostdev' entry: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/> </source> <alias name='hostdev5'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x0f' function='0x0'/> </hostdev> This all works fine during normal operation, but I noticed when we remove the NVMe drive (surprise hotplug event), the PCIe EP then seems "stuck"... here we see the link-down event on the host (when the drive is removed): [67720.177959] pciehp 0000:40:01.2:pcie004: Slot(238-1): Link Down [67720.178027] vfio-pci 0000:42:00.0: Relaying device request to user (#0) And naturally, inside of the Linux VM, we see the NVMe controller drop: [ 1203.491536] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 1203.522759] blk_update_request: I/O error, dev nvme1n2, sector 33554304 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 1203.560505] nvme 0000:01:0f.0: Refused to change power state, currently in D3 [ 1203.561104] nvme nvme1: Removing after probe failure status: -19 [ 1203.583506] Buffer I/O error on dev nvme1n2, logical block 4194288, async page read [ 1203.583514] blk_update_request: I/O error, dev nvme1n1, sector 33554304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 We see this EP is found at IOMMU group '76': # readlink /sys/bus/pci/devices/0000\:42\:00.0/iommu_group ../../../../kernel/iommu_groups/76 And it is no longer bound to the 'vfio-pci' driver (expected) on the host. I was expecting to see all of the FD's to the /dev/vfio/NN character devices closed, but it seems they are still open: # lsof | grep "vfio/76" qemu-kvm 242364 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242502 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242511 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242518 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242531 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242533 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242542 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242550 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242554 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 SPICE 242364 242559 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 After the NVMe drive was removed for 100 seconds, we see the following kernel messages on the host: [67820.179749] vfio-pci 0000:42:00.0: Relaying device request to user (#10) [67900.272468] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars [67900.272652] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars [67900.319284] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars I also noticed these messages related to the EP that is down currently that seem to continue indefinitely on the host (every 100 seconds): [67920.181882] vfio-pci 0000:42:00.0: Relaying device request to user (#20) [68020.184945] vfio-pci 0000:42:00.0: Relaying device request to user (#30) [68120.188209] vfio-pci 0000:42:00.0: Relaying device request to user (#40) [68220.190397] vfio-pci 0000:42:00.0: Relaying device request to user (#50) [68320.192575] vfio-pci 0000:42:00.0: Relaying device request to user (#60) But perhaps that is expected behavior. In any case, the problem comes when I re-insert the NVMe drive into the system... on the host, we see the link-up event: [68418.595101] pciehp 0000:40:01.2:pcie004: Slot(238-1): Link Up But the device is not bound to the 'vfio-pci' driver: # ls -ltr /sys/bus/pci/devices/0000\:42\:00.0/driver ls: cannot access /sys/bus/pci/devices/0000:42:00.0/driver: No such file or directory And appears to fail when attempting to bind to it manually: # echo "0000:42:00.0" > /sys/bus/pci/drivers/vfio-pci/bind -bash: echo: write error: No such device Device is enabled: # cat /sys/bus/pci/devices/0000\:42\:00.0/enable 1 So, wondering if this is expected behavior? Stopping the VM and starting it (virsh destroy/start) allows the device to work in the VM again, but for my particular use case, this is not an option. Need the surprise hotplug functionality to work with the PCIe EP passed into the VM. And perhaps this is an issue elsewhere (eg, vfio-pci). Any tips/suggestions on where to dig more would be appreciated. Thanks for your time. --Marc

2 2