Re: issues with vm after upgrade

On 8/20/21 12:07 PM, daggs wrote: But I didn't see it in the XML diffs that you had posted. That's not exactly true - the bus "number" in libvirt isn't given to qemu as an actual number, but as an alphanumeric device id (called "alias name" in libvirt XML). QEMU doesn't have any concept of "bus number", because (afaiu) there is no way to convey such info to the guest firmware/OS; instead, QEMU creates a topology of interconnected controllers, the firmware and/or OS traverses this topology and assigns numbers to the encountered controllers as it sees fit. So you may have PCI controllers with indexes 1, 2, and 3 in your libvirt config, but those will be described on the QEMU commandline as controllers "pcie.1", "pcie.2", and "pcie.3": -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ and when a PCI device is attached to one of these controllers, the QEMU commandline uses the id name of the controller, not a bus number: -device vfio-pci,host=0000:05:00.0,bus=pci.1,multifunction=on,addr=0x0 \ -device vfio-pci,host=0000:05:00.1,id=hostdev1,bus=pci.1,addr=0x0.0x1 \ It is a nice coincidence that the OSes I've seen happen to traverse the PCI topology in a manner that results in the guest OS numbering the buses the same as they are numbered in libvirt XML that has had PCI addresses auto-assigned by libvirt, but it is trivial to make this *not* happen. For example, if you changed the config so that the bus with index='2' (pcie.2) was attached to pcie.0, addr=0x1.0x1 (i.e. change its PCI address to <address type='pci' bus='0' slot='1' function='1'/>" , and the bus with index='3' was attached to pcie.0, addr=0x1.0x2, then the guest would number "pcie.2" as bus 1, and "pcie.1" as bus 2. And of course the guest OS is free to traverse the controller topology in any manner it wants, so the bus numbering in the guest could be different even if the libvirt-generated QEMU commandline was the same. *HOWEVER*, slot (device) and function number are specified on the QEMU commandline numerically, and they will appear in the guest exactly as they are in the libvirt XML. I would be very interested in seeing the libvirt XML, QEMU commandline, and guest-side output of "lspci" for this. I can't think of any way this could happen without a serious bug *somewhere* (or manual intervention in the PCI addresses in the guest). Right. If your guest was defined the first time using a much older libvirt, then devices would have been assigned to an auto-created dmi-to-pci-bridge at that time, and if you don't change (or remove) the PCI addresses of the devices or the bridge, then that will all be maintained whenever you restart the guest, ragardless of libvirt upgrades. But this again points out that the guest-side PCI addresses (which are determined by the PCI addresses in the libvirt config) should not change when upgrading libvirt (NOTE: 1) libvirt will only auto-assign a new PCI address to a device if it doesn't already have a PCI address assigned to it, and 2) libvirt *never* auto-assigns a non-0 function except when adding a pcie-root-port (and in that case it will always first assign something to function 0)) It wasn't a function of a slot that also contains integrated chipset devices though.... Oh, wait. According to the XML you reference down below, it looks like the audio device you're assigning to the guest *is itself* integrated on the chipset of the host, is that right? (It's interesting that this function of slot 1F is apparently in a different IOMMU group than the other functions of slot 1F. I would have guessed they would all be in the same IOMMU group, resulting in an inability to assign this one function to the guest without at least disabling the other devices on slot 1F (by binding them to the vfio-pci driver) This doesn't show any devices assigned to non-0 functions in the guest (which is the part of what you said in previous messages that sounded wrong to me). (except for the SATA controller, which is listed in the libvirt config only for informational purposes, as it is hardcoded into the basic q35 virtual machine and can't be removed). What is does show is that there is a device a 00:1F.3 *on the host* that is being assigned to 08:01.00 (slot 1, function 0 of the pcie-pci-bridge) in the guest. I'm guessing this is the audio device? Also in this version of the XML, there is no longer a dmi-to-pci-bridge, but there is instead a pcie-to-pci-bridge, implying that you've redefined the guest config, resulting in PCI address auto-assignment being re-run (at least relative to the config you referenced last week that had a dmi-to-pci-bridge). It's possible that the audio device's driver just doesn't like the device being on a standard PCI (i.e. non-PCIe) slot in the guest somehow, since it's a chipset-integrated PCIe device on the host. I haven't heard of that being the case in the past, but it's possible. Anyway, at this point I've lost track of all the changes that have happened (your update entailed much more than just updating the libvirt package - your guest config was also changed/redefined) so I don't know how much more effort should be expended with post-mortem, especially since you now have it working. One thing that I would note is that we should probably be auto-assigning integrated chipset devices to pcie-root-ports rather than to a pci-bridge (I thought we already did that, but I can see how we might not).