Re: [libvirt] [RFC PATCH] support multifunction PCI device

On Fri, May 13, 2011 at 11:32:43AM +0900, KAMEZAWA Hiroyuki wrote: > On Wed, 11 May 2011 08:55:56 +0100 > "Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: > > > On Wed, May 11, 2011 at 10:30:43AM +0800, Wen Congyang wrote: > > > > > > > 3. After hot plug multi function PCI devices: > > > # lspci > > > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) > > > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] > > > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] > > > 00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) > > > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) > > > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 > > > 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) > > > 00:04.0 RAM memory: Red Hat, Inc Virtio memory balloon > > > 00:05.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.0 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.1 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.2 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.3 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.4 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.5 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.6 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > 00:06.7 SCSI storage controller: LSI Logic / Symbios Logic 53c895a > > > > > > > > > > > > > >> > > > >> 2. support to hot unplug multi function PCI device > > > >> hot unpluging multi function PCI device meas that all the other > > > >> devices on the same slot will be hot unpluged. So we should do > > > >> some cleanup and remove the other devices too. If the other > > > >> device on the same slot does not support hot unpluged, or it is a > > > >> a controller and some other devices still use this controller, > > > >> I think we should refuse to hot unplug this mutlti function PCI > > > >> device. > > > > > > > > IMHO these kind of restrictions will make life really unpleasant > > > > for applications and are a reason we should *not* support the > > > > multifunction code. Instead we should focus on one of the other > > > > 3 options I mention above. > > > > > Hmm, how about adding <unpluggable> attribute to <device> definitions ? > IIUC, > 1. There are some unpluggable default devices. This is a little complex. A PCI card can have a boolean unpluggable attribute. A libvirt device can have a tri-state though, unpluggable, not unpluggable, or unpluggable only if all these other functions are unplugged at the same time. May be the third case is actually just the same as the first case here and we should ignore it, and just have a <unpluggable> wrt to the PCI device as a whole. > 2. There are devices which never should be removed by mistake, as rootfs. What devices are considered to be in list 2, can only be determined by the guest OS, so I don't think we can represent that in the libvirt XML. It is also a little bit more fuzzy that a boolean unpluggable. eg from the guest OS POV, no device is unpluggable if it is in use. A block device becomes unpluggable, once the guest filesystem is unmounted. A network card becomes unpluggable, once the network interface is taken offline. It could be desirable to expose this information to management apps, but this won't be possible until QEMU gets some kind of guest agent that can report this info from the guest OS. I think it needs to be reported separately from whether the physical PCI card is unpluggable or not.

> When I've google'd "how to handle 100+ nics with KVM", a qemu community guy > answers "please use multifunction devices" but I disappointed to know > libvirt doesn't support it, now. Ok, I think it is reasonable to support multifunction devices in libvirt, but only if the mgmt app explicitly configures them via the XML. ie, the default behaviour where libvirt auto-assigns PCI addresses should always be to assign slots. And when we get support for many PCI domains or bridges, we'll assign slots from extra domains/bridges too by default. The mgmt app could explicitly override this and configure multifunction by providing an <address> element for devices in the XML, with the function number set to non-zero.