At 05/11/2011 03:55 PM, Daniel P. Berrange Write:
On Wed, May 11, 2011 at 10:30:43AM +0800, Wen Congyang wrote:
At 05/10/2011 06:00 PM, Daniel P. Berrange Write:
On Tue, May 10, 2011 at 02:00:17PM +0800, Wen Congyang wrote:
We want to use more than 200+ device. Libvirt does not use multi function PCI device and PCI-to-PCI bridge. So we can not use more than 200+ device if it's a PCI device or it's controller is a PCI device.
IMHO using multi function support is a dead end, because it makes hotplug completely unusable and is not even possible for most PCI devices. The way we want to raise the device limit is
a. Supporting multiple PCI domains (multiple PCI root complexes)
IIUC, qemu do not support multiple PCI domains now.
There are people who are working on making that supported in KVM real soon.
Good news.
b. Adding PCI bridges
I do not know whether qemu supports it. I find hw/pci_bridge.c in qemu's source, so qemu may support it. I will confirm it.
I'm told this is already supported, but I've not tested it myself.
I'm still reading the qemu's code. Do you know the driver's name(For example: the driver for scsi controller in qemu is lsi.)
c. A virtio-scsi controller to allow > 1 virtio disk per PCI device
Hmm, do you mean this: -device virtio-blk-pci,ports=4,drive0=hda,drive1=hdb,drive2=hdc,...
No it is actually going to be a new device type, and I'm expecting it will work very like the LSI scsi does in terms of device setup. eg create a controller device, and then create devices for each drive.
-device virtio-scsi-pci,id=vscsi1 -drive file=/some/disk,id=vscsi1.1 -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.1 -drive file=/some/disk,id=vscsi1.2 -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.2 -drive file=/some/disk,id=vscsi1.2 -device virtio-blk-pci,bus=vscsi1,drive=vscsi1.2
any of those will dramatically increase the number of devices we can use without the horrible hotplug problems that multifunction introduces.
This patch adds the support of multi function PCI device. It does not support to hot plug/unplug multi function PCI device.
TODO: 1. support to hot plug multi function PCI device We only can hot plug one device at one time. I think we should introduce another command 'virsh attach-devices XXX' to support hot plug more than one device at one time?
Since you can't do practical hotplug of multifunction devices at the QEMU layer at all, there's nothing useful we can do at libvirt either.
We can hotplug multifunction PCI devices like this: 1. Before hot pluging, # lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
2. hot plug multi function PCI devices: # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi2,bus=pci.0,addr=0x06.0x07' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi3,bus=pci.0,addr=0x06.0x06' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi4,bus=pci.0,addr=0x06.0x05' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi5,bus=pci.0,addr=0x06.0x04' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi6,bus=pci.0,addr=0x06.0x03' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi7,bus=pci.0,addr=0x06.0x02' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi8,bus=pci.0,addr=0x06.0x01' # virsh qemu-monitor-command vm1 --hmp 'device_add lsi,id=scsi9,multifunction=on,bus=pci.0,addr=0x06.0x00'
Hmm, that's kinda wierd & i'm suprised it works, particularly for LSI since I thought guest drivers would need support for multifunction too.
I only test LSI. If we can hotplug the device by single PCI device, I think it can work fine for multi function.
3. After hot plug multi function PCI devices: # lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a 00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a
2. support to hot unplug multi function PCI device hot unpluging multi function PCI device meas that all the other devices on the same slot will be hot unpluged. So we should do some cleanup and remove the other devices too. If the other device on the same slot does not support hot unpluged, or it is a a controller and some other devices still use this controller, I think we should refuse to hot unplug this mutlti function PCI device.
IMHO these kind of restrictions will make life really unpleasant for applications and are a reason we should *not* support the multifunction code. Instead we should focus on one of the other 3 options I mention above.
Yes, there's too many restrictions about hot plug/unplug multifunction PCI devices.
Regards, Daniel