On Tue, Sep 20, 2016 at 10:47:53AM +0100, Daniel P. Berrange wrote:
On Tue, Sep 20, 2016 at 02:05:52AM +0530, Kirti Wankhede wrote:
Hi libvirt experts,
Thanks for valuable input on v1 version of RFC.
Quick brief, VFIO based mediated device framework provides a way to virtualize their devices without SR-IOV, like NVIDIA vGPU, Intel KVMGT and IBM's channel IO. This framework reuses VFIO APIs for all the functionalities for mediated devices which are currently being used for pass through devices. This framework introduces a set of new sysfs files for device creation and its life cycle management.
Here is the summary of discussion on v1: 1. Discover mediated device: As part of physical device initialization process, vendor driver will register their physical devices, which will be used to create virtual device (mediated device, aka mdev) to the mediated framework.
Vendor driver should specify mdev_supported_types in directory format. This format is class based, for example, display class directory format should be as below. We need to define such set for each class of devices which would be supported by mediated device framework.
--- mdev_destroy --- mdev_supported_types |-- 11 | |-- create | |-- name | |-- fb_length | |-- resolution | |-- heads | |-- max_instances | |-- params | |-- requires_group |-- 12 | |-- create | |-- name | |-- fb_length | |-- resolution | |-- heads | |-- max_instances | |-- params | |-- requires_group |-- 13 |-- create |-- name |-- fb_length |-- resolution |-- heads |-- max_instances |-- params |-- requires_group
In the above example directory '11' represents a type id of mdev device. 'name', 'fb_length', 'resolution', 'heads', 'max_instance' and 'requires_group' would be Read-Only files that vendor would provide to describe about that type.
'create': Write-only file. Mandatory. Accepts string to create mediated device.
'name': Read-Only file. Mandatory. Returns string, the name of that type id.
Presumably this is a human-targetted title/description of the device.
'fb_length': Read-only file. Mandatory. Returns <number>{K,M,G}, size of framebuffer.
'resolution': Read-Only file. Mandatory. Returns 'hres x vres' format. Maximum supported resolution.
'heads': Read-Only file. Mandatory. Returns integer. Number of maximum heads supported.
None of these should be mandatory as that makes the mdev useless for non-GPU devices.
I'd expect to see a 'class' or 'type' attribute in the directory whcih tells you what kind of mdev it is. A valid 'class' value would be 'gpu'. The fb_length, resolution, and heads parameters would only be mandatory when class==gpu.
Hi Daniel, Here you are proposing to add a class named "gpu", which will make all those gpu related attributes mandatory, which libvirt can allow user to better parse/present a particular mdev configuration? I am just wondering if there is another option that we just make all those attributes that a mdev device can have as optional but still meaningful to libvirt, so libvirt can still parse / recognize them as an class "mdev". In general, I am just trying to understand the requirement from libvirt and see how we can fit in this requirement for both Intel and NVIDIA since Intel is also moving to the type-based interface although they don't have "class" concept yet. Thanks, Neo
'max_instance': Read-Only file. Mandatory. Returns integer. Returns maximum mdev device could be created at the moment when this file is read. This count would be updated by vendor driver. Before creating mdev device of this type, check if max_instance is > 0.
'params' Write-Only file. Optional. String input. Libvirt would pass the string given in XML file to this file and then create mdev device. Set empty string to clear params. For example, set parameter 'frame_rate_limiter=0' to disable frame rate limiter for performance benchmarking, then create device of type 11. The device created would have that parameter set by vendor driver.
Nope, libvirt will explicitly *NEVER* allow arbitrary opaque passthrough of vendor specific data in this way.
The parent device would look like:
<device> <name>pci_0000_86_00_0</name> <capability type='pci'> <domain>0</domain> <bus>134</bus> <slot>0</slot> <function>0</function> <capability type='mdev'> <!-- one type element per sysfs directory --> <type id='11'> <!-- one element per sysfs file roughly --> <name>GRID M60-0B</name> <attribute name='fb_length'>512M</attribute> <attribute name='resolution'>2560x1600</attribute> <attribute name='heads'>2</attribute> <attribute name='max_instances'>16</attribute> <attribute name='requires_group'>1</attribute> </type>
There would need to be a <class> element, eg <class>gpu</class>
We would then have further elements based on the class. eg
<type id='11'> <!-- one element per sysfs file roughly --> <name>GRID M60-0B</name> <fb_length>512M</fb_length> <resolution>2560x1600</resolution> <heads>2</heads> <max_instances>16</max_instances> <requires_group>1</requires_group> </type>
</capability> <product id='...'>GRID M60</product> <vendor id='0x10de'>NVIDIA</vendor> </capability> </device>
2. Create/destroy mediated device
With above example, vGPU device XML would look like:
<device> <name>my-vgpu</name> <parent>pci_0000_86_00_0</parent> <capability type='mdev'> <type id='11'/> <group>1</group> <params>'frame_rate_limiter=0'</params>
No, we will not support <params> in this manner in libvirt.
The entire purpose of libvirt is to represent data in a vendor agnostic manner and not do abitrary passthrough of vendor specific data. Simply saying this field is optional does not get around that either.
</capability> </device>
'type id' is mandatory. 'group' is optional. It should be a unique number in the system among all the groups created for mdev devices. Its usage is: - not needed if single vGPU device is being assigned to a domain. - only need to be set if multiple vGPUs need to be assigned to a domain and vendor driver have 'requires_group' file in type id directory. - if type id directory include 'requires_group' and user tries to assign multiple vGPUs to a domain without having <group> field in XML, it will create single vGPU.
'params' is optional field. User should set this field if extra parameters need to be set for a particular vGPU device. Libvirt don't need to parse these params. These are meant for vendor driver.
Libvirt need to follow the sequence to create device: * Read /sys/../0000\:86\:00.0/11/max_instances. If it is greater than 0, then only proceed else fail.
* Set extra params if 'params' field exist in device XML and 'params' file exist in type id directory
echo "frame_rate_limiter=0" > /sys/../0000\:86\:00.0/11/params
We cannot do that step.
* Autogenerate UUID * Create device:
echo "$UUID:<group>" > /sys/../0000\:86\:00.0/11/create
where <group> is optional. Group should be unique number among all the groups created for mdev devices.
* Clear params, if set earlier:
echo "" > /sys/../0000\:86\:00.0/11/params
* To destroy device:
echo $UUID > /sys/../0000\:86\:00.0/mdev_destroy
3. Start/stop mediated device
No change or requirement for libvirt as this will be handled by open() and close() callbacks to vendor driver. In case of multiple devices and 'requires_group' set, this will be handled in 'first open()' and 'last close()' on device in that group.
4. Launch QEMU/VM
Pass the mdev sysfs path to QEMU as vfio-pci device. For above vGPU device example:
-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID
5. QEMU/VM Shutdown sequence
No change or requirement for libvirt.
6. VM Reset
No change or requirement for libvirt as this will be handled via VFIO reset API and QEMU process will keep running as before.
7. Hot-plug
It is same syntax to create a virtual device for hot-plug.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|