9:30 p.m.
At 05/10/2011 06:00 PM, Daniel P. Berrange Write:
On Tue, May 10, 2011 at 02:00:17PM +0800, Wen Congyang wrote:
> We want to use more than 200+ device. Libvirt does not use multi
> function PCI device and PCI-to-PCI bridge. So we can not use more
> than 200+ device if it's a PCI device or it's controller is a PCI
> device.
IMHO using multi function support is a dead end, because it
makes hotplug completely unusable and is not even possible
for most PCI devices. The way we want to raise the device
limit is
a. Supporting multiple PCI domains (multiple PCI root complexes)
IIUC, qemu do not support multiple PCI domains now.
b. Adding PCI bridges
I do not know whether qemu supports it. I find hw/pci_bridge.c in qemu's source,
so qemu may support it. I will confirm it.
c. A virtio-scsi controller to allow > 1 virtio disk per PCI
device
Hmm, do you mean this:
-device virtio-blk-pci,ports=4,drive0=hda,drive1=hdb,drive2=hdc,...
any of those will dramatically increase the number of devices
we can use without the horrible hotplug problems that multifunction
introduces.
> This patch adds the support of multi function PCI device. It
> does not support to hot plug/unplug multi function PCI device.
>
> TODO:
> 1. support to hot plug multi function PCI device
> We only can hot plug one device at one time. I think we should
> introduce another command 'virsh attach-devices XXX' to support
> hot plug more than one device at one time?
Since you can't do practical hotplug of multifunction devices at the
QEMU layer at all, there's nothing useful we can do at libvirt either.
We can hotplug multifunction PCI devices like this:
1. Before hot pluging,
# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev
20)
00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon
00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
2. hot plug multi function PCI devices:
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi2,bus=pci.0,addr=0x06.0x07'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi3,bus=pci.0,addr=0x06.0x06'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi4,bus=pci.0,addr=0x06.0x05'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi5,bus=pci.0,addr=0x06.0x04'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi6,bus=pci.0,addr=0x06.0x03'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi7,bus=pci.0,addr=0x06.0x02'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi8,bus=pci.0,addr=0x06.0x01'
# virsh qemu-monitor-command vm1 --hmp 'device_add
lsi,id=scsi9,multifunction=on,bus=pci.0,addr=0x06.0x00'
3. After hot plug multi function PCI devices:
# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev
20)
00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon
00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
>
> 2. support to hot unplug multi function PCI device
> hot unpluging multi function PCI device meas that all the other
> devices on the same slot will be hot unpluged. So we should do
> some cleanup and remove the other devices too. If the other
> device on the same slot does not support hot unpluged, or it is a
> a controller and some other devices still use this controller,
> I think we should refuse to hot unplug this mutlti function PCI
> device.
IMHO these kind of restrictions will make life really unpleasant
for applications and are a reason we should *not* support the
multifunction code. Instead we should focus on one of the other
3 options I mention above.
Yes, there's too many restrictions about hot plug/unplug multifunction
PCI devices.
Daniel