4:51 p.m.
Hi
We have a chassis with a peripheral PCI device installed. We have a hypervisor running on
the chassis where we have deployed a VM which can use the PCI device once it is attached.
When the PCI device is powered on or off we need to do a hotplug in/out using virsh
commands. Even though the virsh commands for hotplug is executed successfully, the VM sees
the PCI inside the VM for one or 2 secs. What do you think is the issue?
[root ~]# uname -a
Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Mar 15 05:32:38 UTC 2024 x86_64 x86_64 x86_64
GNU/Linux
[root ~]# virsh --version
8.0.0
[root ~]# rpm -qa | grep libvirt
libvirt-daemon-driver-nwfilter-8.0.0-5.el8.x86_64
python3-libvirt-7.8.0-1.el8.x86_64
libvirt-daemon-driver-storage-logical-8.0.0-5.el8.x86_64
libvirt-libs-8.0.0-5.el8.x86_64
libvirt-daemon-config-nwfilter-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-gluster-8.0.0-5.el8.x86_64
libvirt-8.0.0-5.el8.x86_64
libvirt-daemon-8.0.0-5.el8.x86_64
libvirt-daemon-driver-nodedev-8.0.0-5.el8.x86_64
libvirt-daemon-config-network-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-iscsi-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-rbd-8.0.0-5.el8.x86_64
libvirt-daemon-driver-network-8.0.0-5.el8.x86_64
libvirt-daemon-driver-secret-8.0.0-5.el8.x86_64
python2-libvirt-python-5.10.0-1.el8.x86_64
libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-core-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-iscsi-direct-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-scsi-8.0.0-5.el8.x86_64
libvirt-client-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-disk-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-8.0.0-5.el8.x86_64
libvirt-daemon-driver-interface-8.0.0-5.el8.x86_64
libvirt-daemon-driver-storage-mpath-8.0.0-5.el8.x86_64
libvirt-daemon-kvm-8.0.0-5.el8.x86_64
[root ~]# rpm -qa | grep qemu
qemu-kvm-ui-opengl-6.2.0-11.el8.x86_64
qemu-kvm-6.2.0-11.el8.x86_64
qemu-img-6.2.0-11.el8.x86_64
qemu-kvm-block-iscsi-6.2.0-11.el8.x86_64
ipxe-roms-qemu-20200823-7.git4bd064de.el8.noarch
qemu-kvm-block-gluster-6.2.0-11.el8.x86_64
qemu-kvm-block-rbd-6.2.0-11.el8.x86_64
qemu-kvm-block-curl-6.2.0-11.el8.x86_64
qemu-kvm-core-6.2.0-11.el8.x86_64
qemu-kvm-hw-usbredir-6.2.0-11.el8.x86_64
libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64
qemu-kvm-ui-spice-6.2.0-11.el8.x86_64
qemu-kvm-docs-6.2.0-11.el8.x86_64
qemu-kvm-block-ssh-6.2.0-11.el8.x86_64
qemu-kvm-common-6.2.0-11.el8.x86_64
[root ~]# virsh nodedev-dettach pci_0000_04_00_0
[12013.987821] pci_probe_reset_slot: call pci_slot_reset with probe=1
[12014.063669] pci_slot_reset (printk info): reset hotplug slot.
[12014.134164] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[Mar19 21:09] pci_probe_reset_slot: call pci_slot_reset with probe=1
[ +0.075848] pci_slot_reset (printk info): reset hotplug slot.
[ +0.070495] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
Device pci_0000_04_00_0 detached
[root ~]# virsh attach-device ROUTER8 /opt/us/bin/mrvl.xml
[12024.217540] pci_probe_reset_slot: call pci_slot_reset with probe=1
[ +10.083376] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.293548]
pci_slot_reset (printk info): reset hotplug slot.
[12024.434828] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[ +0.076008] pci_slot_reset (printk info): reset hotplug slot.
[ +0.141280] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)4.526807]
pci_probe_reset_slot: call pci_slot_reset with probe=1
[12024.749367] pci_slot_reset (printk info): reset hotplug slot.
[12024.821327] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[ +0.091979] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.911065]
pci_probe_reset_slot: call pci_slot_reset with probe=1
[ +0.222560] pci_slot_reset (printk info): reset hotplug slot.
[ +0.071960] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
5.057584] pci_slot_reset (printk info): reset hotplug slot.
[ +0.089738] pci_probe_reset_slot: call pci_slot_reset with probe=1
[12025.278585] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[ +0.146519] pci_slot_reset (printk info): reset hotplug slot.
[ +0.221001] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
5.441574] pci_probe_reset_slot: call pci_slot_reset with probe=1
[ +0.162989] pci_probe_reset_slot: call pci_slot_reset with probe=1[12025.665968]
pci_slot_reset (printk info): reset hotplug slot.
[12025.806751] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[12025.898060] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.
[ +0.224394] pci_slot_reset (printk info): reset hotplug slot.
[ +0.140783] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)5.988477]
pci_slot_reset (printk info): reset hotplug slot.
[ +0.091309] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.
12026.205654] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[12026.383424] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset
hotplug slot.
[ +0.090417] pci_slot_reset (printk info): reset hotplug slot.
[ +0.217177] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)6.490406]
pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0)
[ +0.177770] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset
hotplug slot.
2026.726579] pciehp_reset_slot: SLOTCTRL 58 write cmd 0
[ +0.106982] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0)
[ +0.236167] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 0
[ +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 0
[12027.744480] pending interrupts 0x0108 from Slot Status
[ +1.017890] pcieport 0000:00:0e.0: pciehp: pending interrupts 0x0108 from Slot Status
[ +0.000011] pending interrupts 0x0108 from Slot Status
[12027.936469] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008
[ +0.191983] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd
100812028.027182] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down
[12028.190357] Slot(4): Link Down
[ +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008[12028.230605] pcieport
0000:00:0e.0: pciehp: Slot(4): Card not present
[12028.369819] Slot(4): Card not present
Device attached successfully
[12028.415302] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00
[ +0.090713] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down[12028.524356] vfio-pci
0000:04:00.0: Relaying device request to user (#0)
[ +0.163175] Slot(4): Link Down
[ +0.040248] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present
[ +0.139214] Slot(4): Card not present
[ +0.045479] pcieport 0000:00:0e.0: pciehp: pciehp_unconfigure_device: domain:bus:dev =
0000:04:00
[ +0.000004] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00
[ +0.109054] vfio-pci 0000:04:00.0: Relaying device request to user (#0)
9119] pci_probe_reset_slot: call pci_slot_reset with probe=1
[root@nfvis ~]# [12029.151422] pci_slot_reset (printk info): reset hotplug slot.
[12029.237806] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[ +0.334763] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.327572]
pci_probe_reset_slot: call pci_slot_reset with probe=1
[ +0.292303] pci_slot_reset (printk info): reset hotplug slot.
[ +0.086384] pci_r12029.474447] pci_slot_reset (printk info): reset hotplug slot.
eset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[12029.632533] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[ +0.089766] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.788423]
__pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.
[ +0.146875] pci_slot_reset (printk info): reset hotplug slot.
[12029.950782] pci_slot_reset (printk info): reset hotplug slot.
[ +0.158086] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[ +0.155890] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.
.089787] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)
[12030.346708] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset
hotplug slot.
[ +0.162359] pci_slot_reset (printk info): reset hotplug slot.[12030.451555] vfio-pci
0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible)
[ +0.139005] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe =
1)12030.632974] pci 0000:04:00.0: Removing from iommu group 58
[ +0.256921] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset
hotplug slot.
[ +0.104847] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config
space inaccessible)
5279] pciehp_check_link_active: lnk_status = 7011
[ +0.181419] pci 0000:04:00.0: Removing from iommu group 58
[12031.056747] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present
[12031.194871] Slot(4): Card present
[ +0.152301] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_active: lnk_status =
701112031.236173] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up
[12031.395181] Slot(4): Link Up
[ +0.000004] pciehp_check_link_active: lnk_status = 7011
[ +0.271468] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present
[ +0.138124] Slot(4): Card present
[ +0.041302] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up
[ +0.159008] Slot(4): Link Up
31.560448] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id =
0xE61E11AB
[12031.803909] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id =
0xE61E11AB
[12031.909784] pciehp_check_link_status: lnk_status = 7011
[ +0.165267] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id =
0xE61E11AB2031.974018] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor
id = 0xE61E11AB
[12032.176762] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id =
0xE61E11AB
[ +0.243461] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id =
0xE61E11AB
[ +0.105871] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_status: lnk_status =
7011282681] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000
[ +0.000004] pciehp_check_link_status: lnk_status = 7011
[12032.545840] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref]
[12032.701716] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref]
[ +0.064234] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id =
0xE61E11AB2032.794062] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit
pref]
[12032.984416] pci 0000:04:00.0: supports D1 D2
[ +0.202744] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id =
0xE61E11AB2033.039350] pci 0000:04:00.0: Adding to iommu group 58
[12033.201376] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit
pref]
[12033.299989] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit
pref]
[ +0.105919] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000[12033.400664] pci
0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref]
[ +0.263159] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit
pref]12033.568019] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06]
[12033.725967] pcieport 0000:00:0e.0: bridge window [io 0xd000-0xdfff]
[ +0.155876] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit
pref]12033.805813] pcieport 0000:00:0e.0: bridge window [mem 0xdc000000-0xdcffffff]
[12033.982533] pcieport 0000:00:0e.0: bridge window [mem 0x7b30000000-0x7b6fffffff 64bit
pref]
[ +0.092346] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit
pref]12034.086495] pci-stub 0000:04:00.0: claimed by stub
[ +0.190354] pci 0000:04:00.0: supports D1 D2
[ +0.054934] pci 0000:04:00.0: Adding to iommu group 58
[ +0.162026] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit
pref]
[ +0.098613] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit
pref]
[ +0.100675] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit
pref]
[ +0.167355] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06]
[ +0.157948] pcieport 0000:00:0e.0: bridge window [io 0xd000-0xdfff]
[ +0.079846] pcieport 0000:00:0e.0: bridge window [mem 0xdc000000-0xdcffffff]
[ +0.176720] pcieport 0000:00:0e.0: bridge window [mem 0x7b30000000-0x7b6fffffff 64bit
pref]
[ +0.103962] pci-stub 0000:04:00.0: claimed by stub
Thank you for replying in advance,
-Chanda