Re: [libvirt] [Qemu-devel] pci-assign fails with read error on config-space file

On 10/28/16 13:28, Henning Schild wrote: > Hey, > > i am running an unusual setup where i assign pci devices behind the > back of libvirt. I have two options to do that: > 1. a wrapper script for qemu that takes care of suid-root and > appends arguments for pci-assign > 2. virsh qemu-monitor-command ... 'device_add pci-assign...' > > I know i should probably not be doing this, it is a workaround to > introduce fine-grained pci-assignment in an openstack setup, where > vendor and device id are not enough to pick the right device for a > vm. (1) The libvirt domain XML identifies the host PCI device to assign by full PCI address (see the <source> element: <http://libvirt.org/formatdomain.html#elementsHostDev>); it does not filter with vendor/device ID. So, I believe your comment refers to the pci-stub host kernel driver not being flexible enough for binding vs. not binding different instances of the same vendor/device ID.

If that's the case, would you be helped by the following host kernel patch? [PATCH] PCI: pci-stub: accept exceptions to the ID- and class-based matching <http://www.spinics.net/lists/linux-pci/msg55497.html> (2) Is there any reason (other than (1)) that you are using the legacy / deprecated pci-assign method, rather than VFIO? I suggest to evaluate whether the "pci-stub.except=..." kernel parameter helped your use case, and if (consequently) you could move to a fully libvirt + VFIO based config.

Thanks Laszlo > > In both cases qemu will crash with the following output: > >> qemu: hardware error: pci read failed, ret = 0 errno = 22 > > followed by the usual machine state dump. With strace i found it to > be a failing read on the config space file of my device. > /sys/bus/pci/devices/0000:xx:xx.x/config > A few reads out of that file succeeded, as well as accesses on > vendor etc. > > Manually launching a qemu with the pci-assign works without a > problem, so i "blame" libvirt and the cgroup environment the qemu > ends up in. So i put a bash into the exact same cgroup setup - next > to a running qemu, expecting a dd or hexdump on the config-space > file to fail. But from that bash i can read the file without a > problem. > > Has anyone seen that problem before? Right now i do not know what i > am missing, maybe qemu is hitting some limits configured for the > cgroups or whatever. I can not use pci-assign from libvirt, but if i > did would it configure cgroups in a different way or relax some > limits? > > What would be a good next step to debug that? Right now i am > looking at kernel event traces, but the machine is pretty big and > so is the trace. > > That assignment used to work and i do not know how it broke, i have > tried combinations of several kernels, versions of libvirt and qemu. > (kernel 3.18 and 4.4, libvirt 1.3.2 and 2.0.0, and qemu 2.2.1 and > 2.7) All combinations show the same problem, even the ones that > work on other machines. So when it comes to software versions the > problem could well be caused by a software update of another > component, that i got with the package manager and did not compile > myself. It is a debian 8.6 with all recent updates installed. My > guess would be that systemd could have an influence on cgroups or > limits causing such a problem. > > regards, > Henning >