- Users - libvirt List Archives

PCI Passthrough and Surprise Hotplug
by Marc Smith 08 Oct '20

08 Oct '20

Hi, I'm using QEMU/KVM on RHEL (CentOS) 7.8.2003: # cat /etc/redhat-release CentOS Linux release 7.8.2003 I'm passing an NVMe drive into a Linux KVM virtual machine (<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>) which has the following 'hostdev' entry: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/> </source> <alias name='hostdev5'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x0f' function='0x0'/> </hostdev> This all works fine during normal operation, but I noticed when we remove the NVMe drive (surprise hotplug event), the PCIe EP then seems "stuck"... here we see the link-down event on the host (when the drive is removed): [67720.177959] pciehp 0000:40:01.2:pcie004: Slot(238-1): Link Down [67720.178027] vfio-pci 0000:42:00.0: Relaying device request to user (#0) And naturally, inside of the Linux VM, we see the NVMe controller drop: [ 1203.491536] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 1203.522759] blk_update_request: I/O error, dev nvme1n2, sector 33554304 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 1203.560505] nvme 0000:01:0f.0: Refused to change power state, currently in D3 [ 1203.561104] nvme nvme1: Removing after probe failure status: -19 [ 1203.583506] Buffer I/O error on dev nvme1n2, logical block 4194288, async page read [ 1203.583514] blk_update_request: I/O error, dev nvme1n1, sector 33554304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 We see this EP is found at IOMMU group '76': # readlink /sys/bus/pci/devices/0000\:42\:00.0/iommu_group ../../../../kernel/iommu_groups/76 And it is no longer bound to the 'vfio-pci' driver (expected) on the host. I was expecting to see all of the FD's to the /dev/vfio/NN character devices closed, but it seems they are still open: # lsof | grep "vfio/76" qemu-kvm 242364 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242502 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242511 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242518 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242531 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242533 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242542 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242550 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 qemu-kvm 242364 242554 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 SPICE 242364 242559 qemu 70u CHR 235,4 0t0 3925324 /dev/vfio/76 After the NVMe drive was removed for 100 seconds, we see the following kernel messages on the host: [67820.179749] vfio-pci 0000:42:00.0: Relaying device request to user (#10) [67900.272468] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars [67900.272652] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars [67900.319284] vfio_bar_restore: 0000:42:00.0 reset recovery - restoring bars I also noticed these messages related to the EP that is down currently that seem to continue indefinitely on the host (every 100 seconds): [67920.181882] vfio-pci 0000:42:00.0: Relaying device request to user (#20) [68020.184945] vfio-pci 0000:42:00.0: Relaying device request to user (#30) [68120.188209] vfio-pci 0000:42:00.0: Relaying device request to user (#40) [68220.190397] vfio-pci 0000:42:00.0: Relaying device request to user (#50) [68320.192575] vfio-pci 0000:42:00.0: Relaying device request to user (#60) But perhaps that is expected behavior. In any case, the problem comes when I re-insert the NVMe drive into the system... on the host, we see the link-up event: [68418.595101] pciehp 0000:40:01.2:pcie004: Slot(238-1): Link Up But the device is not bound to the 'vfio-pci' driver: # ls -ltr /sys/bus/pci/devices/0000\:42\:00.0/driver ls: cannot access /sys/bus/pci/devices/0000:42:00.0/driver: No such file or directory And appears to fail when attempting to bind to it manually: # echo "0000:42:00.0" > /sys/bus/pci/drivers/vfio-pci/bind -bash: echo: write error: No such device Device is enabled: # cat /sys/bus/pci/devices/0000\:42\:00.0/enable 1 So, wondering if this is expected behavior? Stopping the VM and starting it (virsh destroy/start) allows the device to work in the VM again, but for my particular use case, this is not an option. Need the surprise hotplug functionality to work with the PCIe EP passed into the VM. And perhaps this is an issue elsewhere (eg, vfio-pci). Any tips/suggestions on where to dig more would be appreciated. Thanks for your time. --Marc

2 2

[libvirt-users] Live migrate + change interface name
by Marc Roos 08 Oct '20

08 Oct '20

Is it possible to do a live migrate of a guest, having on the from host a source_device=eth2 and to host a source_dev=eth1?

2 4

time in domain very unstable
by Lentes, Bernd 08 Oct '20

08 Oct '20

Hi, i have a domain (SLES 10 SP4) running with KVM. Time is very wrong when booting unless ntp synchronizes. What can i do ? Bernd -- Bernd Lentes Systemadministration Institute for Metabolism and Cell Death (MCD) Building 25 - office 122 HelmholtzZentrum München bernd.lentes(a)helmholtz-muenchen.de phone: +49 89 3187 1241 phone: +49 89 3187 3827 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/mcd stay healthy Helmholtz Zentrum MÃ¼nchen Helmholtz Zentrum MÃ¼nchen

2 3

Unable to start or create any vm's on the qemu/kvm network
by muke101 07 Oct '20

07 Oct '20

Hi, for some reason my libvirt install has suddenly stopped working with any vm's on the qemu/kvm network, and I can't seem to figure out how to fix it or even anything that indicates what's causing the problem. I've posted my problem to discord servers, subreddits, irc's and the server fault stackexchange to no avail, so I'm hoping that if I share my stackexchnage post here someone might be able to take a look and figure out the problem? Here's the post with more details: https://serverfault.com/questions/1036370/error-when-starting-vm-on-the-qem… Any help would be appreciated, thanks.

2 1

what does "virsh destroy" really ?
by Lentes, Bernd 06 Oct '20

06 Oct '20

Hi, what does "virsh destroy" with the domain ? Send a kill -9 to the process ? Bernd -- Bernd Lentes Systemadministration Institute for Metabolism and Cell Death (MCD) Building 25 - office 122 HelmholtzZentrum München bernd.lentes(a)helmholtz-muenchen.de phone: +49 89 3187 1241 phone: +49 89 3187 3827 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/mcd stay healthy Helmholtz Zentrum MÃ¼nchen Helmholtz Zentrum MÃ¼nchen

5 6

Any way to persistently edit a single VM's AppArmor profile?
by Brian Turek 06 Oct '20

06 Oct '20

In order to test a patch I submitted I've been experimenting with "qemu:commandline" to use some newer features for a QEMU host/guest file share. I quickly ran into issues with AppArmor as virt-aa-helper understandably doesn't parse "qemu:commandline" for directories to add to the dynamically generated AppArmor profile. After reading a bunch of documentation, I cannot find a way to persistently edit a single VM's AppArmor profile. virt-aa-helper will respect a pre-existing "/etc/apparmor.d/libvirt/libvirt-<uuid>" file but then delete it when the VM shuts down. virt-aa-helper does not respect pre-existing "/etc/apparmor.d/libvirt/libvirt-<uuid>.files" and will just overwrite it. The best I came up with was to edit "/etc/apparmor.d/abstractions/libvirt-qemu" but that affects ALL QEMU-based VMs whereas I really only need to tweak one profile. I'm an AppArmor novice so I'm hoping there might be some other way to do what I need. Anyone have any ideas? Thank you

1 0

Determining when a guest booted / how long it's been running
by Digimer 02 Oct '20

02 Oct '20

4 6

cant create network with virt-manager
by Marko Horn 29 Sep '20

29 Sep '20

hello list, i cant create an isolated network with the virt-manager. installed version virt-manager 3.0.0 installed version libvirt 6.2.0 output in error-message: Error creating virtual network: internal error: Failed to apply firewall rules /sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr3 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name. Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 66, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/createnet.py", line 428, in _async_net_create netobj.create() File "/usr/lib/python3.7/site-packages/libvirt.py", line 3174, in create if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self) libvirt.libvirtError: internal error: Failed to apply firewall rules /sbin/iptables -w --table filter --insert LIBVIRT_INP --in-interface virbr3 --protocol tcp --destination-port 67 --jump ACCEPT: iptables: No chain/target/match by that name any ideas? thank you marko -- Mitten drin statt nur Datei!

2 2

Help on Meson build Error
by Wei Wang 25 Sep '20

25 Sep '20

Seems it didn't appear on the mailing list, resent it. Hi folks, I'm trying to build libvirt using meson with the latest upstream libvirt, but the compilation fails: (followed on https://libvirt.org/compiling.html, not sure if any dependencies are missed. I checked, src/util/libvirt_util.a.p does not exist.) FAILED: src/util/libvirt_util.a.p/glibcompat.c.o .... In file included from ../src/util/glibcompat.c:19: ./config.h:1026:10: fatal error: config-post.h: No such file or directory FAILED: src/util/libvirt_util.a.p/virfirmware.c.o In file included from ../src/util/virarch.c:22: ./config.h:1026:10: fatal error: config-post.h: No such file or directory FAILED: src/util/libvirt_util.a.p/viraudit.c.o In file included from ../src/util/viraudit.c:22: ./config.h:1026:10: fatal error: config-post.h: No such file or directory Any help is appreciated. Thanks, Wei

4 20

[libvirt] SRIOV configuration
by Edward Haas 25 Sep '20

25 Sep '20

Hello, We encountered a problem while testing SRIOV on kubevirt [1]: We are currently using the Device "hostdev" generic options to pass-through the VF to the guest. It is not done through the "interface" (with type hostdev) because that option requires access to the PF which is not available in our case (the VF is moved to the container namespace where libvirt runs, but not the PF, however libvirt attempts to access the PF and fails). The PCI addresses appearing on the domxml are not the same as the ones mappend/detected in the VM itself. I compared the domxml on the host and the lspci in the VM while the VM runs. This occurs only when SRIOV is defined, messing up also the other "regular" vnics. Somehow, everything comes up and runs (with the SRIOV interface as well) on the first boot (even though the PCI addresses are not in sync), but additional boots cause the VM to mess up the interfaces (not all are detected). This is how the domxml hostdev section looks like: ``` <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x3b' slot='0x0a' function='0x4'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x01' function='0x0'/> </hostdev> ``` Is there something we are missing or we misconfigured? Tested with 6.0.0-16.fc31 My second question is: Can libvirt avoid accessing the PF (as we do not need mac and other options). 1. https://github.com/kubevirt/kubevirt Thanks, Edy.

3 2