Re: [libvirt] [RFC] libvirt vGPU QEMU integration

On 08/18/2016 12:41 PM, Neo Jia wrote: Thanks for the detailed description. This is very helpful. We've discussed this question offline, but I just want to make sure I understood correctly - all initialization of the physical device on the host is already handled "elsewhere", so libvirt doesn't need to be concerned with any physical device lifecycle or configuration (setting up the number or types of vGPUs), correct? Do you think this would also be the case for other vendors using the same APIs? I guess this all comes down to whether or not the setup of the physical device is defined within the bounds of the common infrastructure/API, or if it's something that's assumed to have just magically happened somewhere else. I had originally thought that libvirt would be setting up and managing a pool of virtual devices, similar to what we currently do with SRIOV VFs. But from this it sounds like the management of this pool is completely handled by your drivers (especially since the contents of the pool can apparently completely change at any instant). In one way that makes life easier for libvirt, because it doesn't need to manage anything. On the other hand, it makes thing less predictable. For example, when libvirt defines a domain, it queries the host system to see what types of devices are legal in guests on this host, and expects those devices to be available at a later time. As I understand it (and I may be completely wrong), when no vGPUs are running on the hardware, there is a choice of several different models of vGPU (like the example you give below), but when the first vGPU is started up, that triggers the host driver to restrict the available models. If that's the case, then a particular vGPU could be "available" when a domain is defined, but not an option by the time the domain is started. That's not a show stopper, but I want to make sure I am understanding everything properly. Also, is there any information about the maximum number of vGPUs that can be handled by a particular physical device (I think that changes based on which model of vGPU is being used, right?) Or maybe what is the current "load" on a physical device, in case there is more than one and libvirt (or management) wants to make a decision about which one to use? Is there any reason to try to maintain the same UUID from one run to the next? Or should we completely think of this as a cookie for this time only (so more like a file handle, but we get to pick the value)? (Michal has asked about this in relation to migration, but the question also applies in the general situation of simply stopping and restarting a guest). Also, is it enforced that "UUID" actually be a 128 bit UUID, or can it be any unique string? 1) I have the same question as Michal - you're passing the path to the sysfs directory for the device to qemu, which implies that the qemu process will need to open/close/read/write files in that directory. Since libvirt is running as root, it can easily do that, but libvirt then runs the qemu process under a different uid and with a different selinux context. We need to make sure that we can change the uid/selinux labelling of the items in sysfs without adverse effect elsewhere. Also it's important that qemu doesn't need to access anything else outside of this device-specific directory (each qemu process is running with different selinux labeling and potentially a different uid:gid, so if there is any common file/device node that must be accessed directly by qemu, it would need to be safely globally readable/writable. How does this device show up in the guest? guess it's a PCI device (since you're using vfio-pci :-), and all the standard options for setting PCI address apply. And is this device legacy PCI, or PCI Express? (Or perhaps it changes behavior depending on the type of slot used in the guest?) From what I understand here, it sounds like what's needed from libvirt is 1) exposing enough info in the output of nodedev-dumpxml for an application to use it to determine which devices are capable of creating vGPUs, and which models of vGPU they can create. 2) to create+start (then later stop+destroy) individual vGPUs based on [something] in the domain XML. So the question that remains is how to put it in the domain config. My first instinct was to use some variation of <hostdev> (since the backend of it is vfio-pci), but on the other hand hostdev is usually used to take one device that could be used by the host, take it away from the host, and give it to the guest, and that's not exactly what's happening here. So I wonder if there would be any advantage to making this another model of <video> instead.