10:39 p.m.
Hi Marcel,
> Indeed, if a device is attached to a PCI bus it makes no sense to
advertise the extended configuration space.
> Can you please share the QEMU command line? Maybe is possible to make the
device's bus PCIe in QEMU?
Changing the following should make the tweak I proposed irrelevant.
- device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.2,addr=0x4
+ device vfio-pci,host=03:00.0,id=hostdev0,bus=pcie.0,addr=0x4
Here's the full command for reference.
2016-07-19 18:42:22.110+0000: starting up libvirt version: 1.2.17, package: 13.el7 (CentOS
BuildSystem <http://bugs.centos.org>;, 2015-11-20-16:24:10, worker1.bsys.centos.org),
qemu version: 2.3.0 (qemu-kvm-ev-2.3.0-31.el7_2.10.9)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -name instance-00000019 -S -machine
pc-q35-rhel7.2.0,accel=kvm,usb=off -cpu
Haswell-noTSX,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 8192 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid
8b6865eb-2118-430e-a7e0-2989696576b1 -smbios type=1,manufacturer=Fedora
Project,product=OpenStack
Nova,version=13.1.0-1.el7,serial=a1068a93-24d1-4da2-8903-f9b8307fb0d8,uuid=8b6865eb-2118-430e-a7e0-2989696576b1,family=Virtual
Machine -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000019/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device
i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device
pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -drive
file=/var/lib/nova/instances/8b6865eb-2118-430e-a7e0-2989696576b1/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.2,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/var/lib/nova/instances/8b6865eb-2118-430e-a7e0-2989696576b1/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.2,addr=0x3,drive=drive-virtio-disk1,id=virtio-disk1
-netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:9c:cf:60,bus=pci.2,addr=0x1 -chardev
file,id=charserial0,path=/var/lib/nova/instances/8b6865eb-2118-430e-a7e0-2989696576b1/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device
isa-serial,chardev=charserial1,id=serial1 -vnc 0.0.0.0:0 -k en-us -device
cirrus-vga,id=video0,bus=pcie.0,addr=0x1 -device
vfio-pci,host=03:00.0,id=hostdev0,bus=pci.2,addr=0x4 -device
virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/3 (label charserial1)
It seems very reasonable to change the qemu command line. However, I'm using
OpenStack Nova to launch the instance, which then uses libvirt, which finally uses qemu.
So, the issues is very buried.
I think that any instance of a q35 machine where the assigned device
is
placed on the conventional PCI bridge will create this scenario. It's
the default for attaching devices to a libvirt managed q35 VM AFAIK.
Yes, I discussed with Laine from libvirt the possibility to assign
devices to a PCIe port instead.
Yes, I see this as the issue as well. The VM is being created by libvirt with the
following xml. The problem is that the device is auto-assigned to the pci-bridge by
default.
<domain type="kvm">
<uuid>8b6865eb-2118-430e-a7e0-2989696576b1</uuid>
<name>instance-00000019</name>
<memory>8388608</memory>
<vcpu>8</vcpu>
... <snip> ...
<devices>
... <snip> ...
<hostdev mode="subsystem" type="pci"
managed="yes">
<source>
<address bus="0x03" domain="0x0000"
function="0x0" slot="0x00"/>
</source>
</hostdev>
... <snip> ...
</devices>
</domain>
If I do a dumpxml I get the following
<domain type="kvm">
<uuid>8b6865eb-2118-430e-a7e0-2989696576b1</uuid>
<name>instance-00000019</name>
<memory>8388608</memory>
<vcpu>8</vcpu>
... <snip> ...
<devices>
... <snip> ...
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1'
model='dmi-to-pci-bridge'>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01'
slot='0x01' function='0x0'/>
</controller>
<interface type='bridge'>
... <snip> ...
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x02'
slot='0x02' function='0x0'/>
</hostdev>
</devices>
</domain>
If I manually change the domain as follows
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
</source>
<alias name='hostdev0'/>
- <address type='pci' domain='0x0000' bus='0x02'
slot='0x02' function='0x0'/>
+ <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0'/>
</hostdev>
I can reimport and the device and it attaches successfully to the pcie-root hub. The
problem is I need to either manually specify this in nova or update the behavior of
libvirt to auto assign pcie devices to pcie busses. This would involve reading pci
configuration space to check if the device is a pcie devices. Then attaching to the
pcie-root, if possible.
>>
>> In fact, I've tried to fix this multiple times:
>>
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-10/msg05384.html
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg02422.html
>> https://lists.nongnu.org/archive/html/qemu-devel/2016-01/msg03259.html
>>
>> Yet the patch remains unapplied :(
>
> I thought is it in already. Maybe Michael can add it as part of the hard freeze.
> And if the patch will be applied, the tweak above wouldn't help, right Alex?
The tweak Spenser suggested above should be unnecessary with my proposed
patch applied.
I think I finally understand this. The bus is not pcie -> we
return
from vfio_add_ext_cap without "breaking" the extended capabilities
chain and the bare metal SR-IOV capability will be visible to guest.
With your patch the PCI bridge will "interfere" and mask the extended
configuration space completely.
Only now searching for that patch did I notice Michael's
comment hidden at the bottom of his reply, which I assume is why it
never got applied:
https://patchwork.kernel.org/patch/8057411/
I just saw it too! It seems Michael wants to cache this info
in device instead of re-calculating it every time.
> Anyway, the current behavior is clearly a bug, so QEMU hard
freeze
> should be irrelevant. If anyone wants to take over the patch, feel
> free. Thanks,
I suppose I can handle it, but sadly not for 2.7.
If Spencer has some time now he can help by testing it and reviewing it quickly :)
I'd be happy to help; but that patch really just more permanently breaks my current
workaround ;-) .
Thanks,
Spenser
This email and any attachments are intended for the sole use of the named recipient(s) and
contain(s) confidential information that may be proprietary, privileged or copyrighted
under applicable law. If you are not the intended recipient, do not read, copy, or forward
this email message or any attachments. Delete this email message and any attachments
immediately.
10:51 p.m.
New subject: [libvirt-users] [Qemu-devel] VFIO PCIe Extended Capabilities
On Tue, 19 Jul 2016 20:39:26 +0000
Spenser Gilliland <spenser.gilliland(a)xilinx.com> wrote:
>> Anyway, the current behavior is clearly a bug, so QEMU hard
freeze
>> should be irrelevant. If anyone wants to take over the patch, feel
>> free. Thanks,
> I suppose I can handle it, but sadly not for 2.7.
> If Spencer has some time now he can help by testing it and reviewing it quickly :)
I'd be happy to help; but that patch really just more permanently breaks my current
workaround ;-) .
How so? The patch should make extended config space disappear when the
device is not on a PCIe bus. This is the correct behavior per the
PCIe spec. Are you trying to exploit this QEMU bug to see some PCIe
capabilities but not others? That's wrong, you'll need to move the
device to a PCIe slot to get extended capabilities.
This email and any attachments are intended for the sole use of the
named recipient(s) and contain(s) confidential information that may be proprietary,
privileged or copyrighted under applicable law. If you are not the intended recipient, do
not read, copy, or forward this email message or any attachments. Delete this email
message and any attachments immediately.
You're sending email to a public list, please stop including these.
Thanks,
Alex
11:25 p.m.
New subject: [libvirt-users] [Qemu-devel] VFIO PCIe Extended Capabilities
Hey Alex,
How so? The patch should make extended config space disappear when
the
device is not on a PCIe bus. This is the correct behavior per the
PCIe spec. Are you trying to exploit this QEMU bug to see some PCIe
capabilities but not others? That's wrong, you'll need to move the
device to a PCIe slot to get extended capabilities.
Yes, I am exploiting the peculiarities of QEMU to make my hardware work properly. It may
be wrong but it works (hence the ;-) ) .
From this conversation, it seems the real problem is that libvirt does
not auto assign the hardware to a pcie root port in the q35 machine type. It seems like
this really needs to be fixed in the libvirt source code; so, I'm working on a patch
for libvirt.
I'd be happy to test your patch for you; I have access to hardware that will allow me
to verify that your patch works correctly. Feel free to cc me on the next version of the
patch and I'll add my tested-by.
You're sending email to a public list, please stop including
these.
This is not intentional. It seems like the Xilinx email servers (not my local machine)
are adding this to all messages. I'll see what I can do and I'm sorry for the
inconvenience.
Thanks,
Spenser
This email and any attachments are intended for the sole use of the named recipient(s) and
contain(s) confidential information that may be proprietary, privileged or copyrighted
under applicable law. If you are not the intended recipient, do not read, copy, or forward
this email message or any attachments. Delete this email message and any attachments
immediately.
2981
days inactive
2981
days old
2 comments
2 participants
participants (2)
-
Alex Williamson -
Spenser Gilliland