Re: [libvirt] [RFC] libvirt vGPU QEMU integration

On 18.08.2016 18:41, Neo Jia wrote: > Hi libvirt experts, Hi, welcome to the list. > > I am starting this email thread to discuss the potential solution / proposal of > integrating vGPU support into libvirt for QEMU. > > Some quick background, NVIDIA is implementing a VFIO based mediated device > framework to allow people to virtualize their devices without SR-IOV, for > example NVIDIA vGPU, and Intel KVMGT. Within this framework, we are reusing the > VFIO API to process the memory / interrupt as what QEMU does today with passthru > device. So as far as I understand, this is solely NVIDIA's API and other vendors (e.g. Intel) will use their own or is this a standard that others will comply to?

> > The difference here is that we are introducing a set of new sysfs file for > virtual device discovery and life cycle management due to its virtual nature. > > Here is the summary of the sysfs, when they will be created and how they should > be used: > > 1. Discover mediated device > > As part of physical device initialization process, vendor driver will register > their physical devices, which will be used to create virtual device (mediated > device, aka mdev) to the mediated framework. > > Then, the sysfs file "mdev_supported_types" will be available under the physical > device sysfs, and it will indicate the supported mdev and configuration for this > particular physical device, and the content may change dynamically based on the > system's current configurations, so libvirt needs to query this file every time > before create a mdev. Ah, that was gonna be my question. Because in the example below, you used "echo '...vgpu_type_id=20...' > /sys/bus/.../mdev_create". And I was wondering where does the number 20 come from. Now what I am wondering about is how libvirt should expose these to users. Moreover, how it should let users to chose. We have a node device driver where I guess we could expose possible options and then require some explicit value in the domain XML (but what value would that be? I don't think taking vgpu_type_id-s as they are would be a great idea).

> > Note: different vendors might have their own specific configuration sysfs as > well, if they don't have pre-defined types. > > For example, we have a NVIDIA Tesla M60 on 86:00.0 here registered, and here is > NVIDIA specific configuration on an idle system. > > For example, to query the "mdev_supported_types" on this Tesla M60: > > cat /sys/bus/pci/devices/0000:86:00.0/mdev_supported_types > # vgpu_type_id, vgpu_type, max_instance, num_heads, frl_config, framebuffer, > max_resolution > 11 ,"GRID M60-0B", 16, 2, 45, 512M, 2560x1600 > 12 ,"GRID M60-0Q", 16, 2, 60, 512M, 2560x1600 > 13 ,"GRID M60-1B", 8, 2, 45, 1024M, 2560x1600 > 14 ,"GRID M60-1Q", 8, 2, 60, 1024M, 2560x1600 > 15 ,"GRID M60-2B", 4, 2, 45, 2048M, 2560x1600 > 16 ,"GRID M60-2Q", 4, 4, 60, 2048M, 2560x1600 > 17 ,"GRID M60-4Q", 2, 4, 60, 4096M, 3840x2160 > 18 ,"GRID M60-8Q", 1, 4, 60, 8192M, 3840x2160 > > 2. Create/destroy mediated device > > Two sysfs files are available under the physical device sysfs path : mdev_create > and mdev_destroy > > The syntax of creating a mdev is: > > echo "$mdev_UUID:vendor_specific_argument_list" > > /sys/bus/pci/devices/.../mdev_create > > The syntax of destroying a mdev is: > > echo "$mdev_UUID:vendor_specific_argument_list" > > /sys/bus/pci/devices/.../mdev_destroy > > The $mdev_UUID is a unique identifier for this mdev device to be created, and it > is unique per system. Ah, so a caller (the one doing the echo - e.g. libvirt) can generate their own UUID under which the mdev will be known? I'm asking because of migration - we might want to preserve UUIDs when a domain is migrated to the other side. Speaking of which, is there such limitation or will guest be able to migrate even if UUID's changed?

> > For NVIDIA vGPU, we require a vGPU type identifier (shown as vgpu_type_id in > above Tesla M60 output), and a VM UUID to be passed as > "vendor_specific_argument_list". I understand the need for vgpu_type_id, but can you shed more light on the VM UUID? Why is that required?

> > If there is no vendor specific arguments required, either "$mdev_UUID" or > "$mdev_UUID:" will be acceptable as input syntax for the above two commands. > > To create a M60-4Q device, libvirt needs to do: > > echo "$mdev_UUID:vgpu_type_id=20,vm_uuid=$VM_UUID" > > /sys/bus/pci/devices/0000\:86\:00.0/mdev_create > > Then, you will see a virtual device shows up at: > > /sys/bus/mdev/devices/$mdev_UUID/ > > For NVIDIA, to create multiple virtual devices per VM, it has to be created > upfront before bringing any of them online. > > Regarding error reporting and detection, on failure, write() to sysfs using fd > returns error code, and write to sysfs file through command prompt shows the > string corresponding to error code. > > 3. Start/stop mediated device > > Under the virtual device sysfs, you will see a new "online" sysfs file. > > you can do cat /sys/bus/mdev/devices/$mdev_UUID/online to get the current status > of this virtual device (0 or 1), and to start a virtual device or stop a virtual > device you can do: > > echo "1|0" > /sys/bus/mdev/devices/$mdev_UUID/online > > libvirt needs to query the current state before changing state. > > Note: if you have multiple devices, you need to write to the "online" file > individually. > > For NVIDIA, if there are multiple mdev per VM, libvirt needs to bring all of > them "online" before starting QEMU. This is a valid requirement, indeed.

> > 4. Launch QEMU/VM > > Pass the mdev sysfs path to QEMU as vfio-pci device: > > -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$mdev_UUID,id=vgpu0 One question here. Libvirt allows users to run qemu under different user:group than root:root. If that's the case, libvirt sets security labels on all files qemu can/will touch. Are we going to need to do something in that respect here?

> > 5. Shutdown sequence > > libvirt needs to shutdown the qemu, bring the virtual device offline, then destroy the > virtual device > > 6. VM Reset > > No change or requirement for libvirt as this will be handled via VFIO reset API > and QEMU process will keep running as before. > > 7. Hot-plug > > It optional for vendors to support hot-plug. > > And it is same syntax to create a virtual device for hot-plug. > > For hot-unplug, after executing QEMU monitor "device del" command, libvirt needs > to write to "destroy" sysfs to complete hot-unplug process. > > Since hot-plug is optional, then mdev_create or mdev_destroy operations may > return an error if it is not supported. Thank you for very detailed description! In general, I like the API as it looks usable from my POV (I'm no VFIO devel though). Michal