Hey all! So here I finally have a rebased the HVM USB hotplug series from last year. I went through and addressed all of IanC's technical comments that I could; so it should be in much better shape than it was. However, one of the unfinished threads of the conversation at that time was about the interface. In particular since Simon Bo is now working on the PVUSB side of things, I thought it would be good to review the situation so we can get an interface we're happy with. In order to really answer the questions, I went through and looked at the underlying capabilities of QEMU and PVUSB, at the existing libxl and xend interfaces for those two, and also at the libvirt interface. I summarize what I've found here; but if you want a TLDR version, just skip to the bottom, where it says "Draft Design Proposal". But before asking any questions or making any criticisms, please go back and read the appropriate summary section. References are at the very bottom of the e-mail. This can be found at the following address: git://xenbits.xen.org/people/gdunlap/xen.git out/hvm-usb-hotplug-v7-RFC == Capabilities of QEMU == Qemu has the additional ability to not only pass through host devices, but other kinds of devices -- tablets, mice, keyboards, drives, hubs, and so on. As far as I know, PVUSB can only do host device pass-through at the moment. (Although you could certainly imagine that changing; if qemu was taught how to become a PVUSB back-end, for example.) qemu has two ways of specifying usb devices. The "legacy" method can be specified on the qemu command-line or via the qemu monitor. It is limited to USB 1.1. The new "qdevice" method can be used either on the command-line or via qmp. qemu-traditional only supports the legacy method. qemu-xen still supports the legacy method via command-line, but we should try to start using the qdevice method if we can. qemu-xen has the ability to create virtual USB 2.0 or 3.0 controllers. These can be named, and it is possible to specify (to a certain level) which controller to plug a device into. These can be created either via the command-line, or hot-plugged via qmp. qemu also seems to be willing to shift things around behind the scenes to be able to handle your request; shifting around the USB topology, for instance. It will also do "helpful" things; for instance, if you specify that you want to plug a device into a USB 2.0 controller, and the device is only 1.1 capable, it will ignore what you asked and plug it into the 1.1 controller. == Capabilities of PVUSB == PVUSB also requires you to first create a virtual controller, before attaching host devices to it. There is some flexibility in how to create these, and you can create more than one and specify which virtual bus to attach the host device. You seem to be able to specify the USB version number when you specify the controller as well. You can also specify the number of ports for a device; I'm not sure what the maximum number of ports per controller is, or why you would ever really want to have less than the maximum number of ports. It looks like in the xm interface, when you create a virtual controller it is assigned an index, starting at 0; and this is the index that is used when specifying where to plug in a device. (Simon, can you please correct this if I'm wrong, and add anything important that I missed?) == libxl/xl interface == At the moment, libxl/xl only support USB at domain creation time. For HVM guests, we have two incompatible sets of options. usb=1 and usbdevice=[] use the "old-style" qemu interface on the qemu command-line. The "old-style" interface can only be used to specify USB 1.1 devices. We also have "usbversion=[foo]", which uses the "new-style" device specification commands. These new specification commands *can* be specified either on the qemu command-line or via qmp; but at the moment libxl specifies them on the command-line. The patch series attached specifes using the new-style interface via qmp. (This seems to work properly with the various controllers no matter how they were created.) In theory, we should be able to attach these devices during domain creation, after qemu has been started but before the guest is running. Obviously, we would ideally like for the user not to have to worry about a lot of this complexity, and just say, "Can you pass this device through to the guest? Thanks." Another thing to consider in the design space is config file and start-up behavior. == Libvirt's interface == I've had a brief look at libvirt's USB interface, and learned a bit about libvirt's general approach to things at the Xen Hackathon last week. One of the goals of libvirt is to be able to specify the virtual hardware in enough detail to keep it from changing when you upgrade the hypervisor, so that certain proprietary operating systems which are sensitive to this kind of thing continue to work. Instead of specifying a controller name, you specify a controller index (which is different than qemu). But instead of specifying a USB version number, you specify a model for the USB controller (which happens to be the exact name that qemu uses): "piix3-uhci", "piix4-uhci", "ehci", "ich9-ehci1", "ich9-uhci1", "ich9-uhci2", "ich9-uhci3", "vt82c686b-uhci", "pci-ohci" or "nec-xhci". When specifying a USB device, libvirt has the concept of an "address" where you will plug it into. Here is what the page says about it: "USB addresses have the following additional attributes: bus (a hex value between 0 and 0xfff, inclusive), and port (a dotted notation of up to four octets, such as 1.2 or 2.1.3.1)." It's not exactly clear to me what those numbers mean. But after chatting with Daniel Berrange at the Hackathon, it looks like the "bus" in the address corresponds to the "index" in the controller specification. It also appears that this "index" is internal to libvirt and is not exposed to the guest: so it should in theory be possible to have index 0 be a Xen PVUSB controller, 1 be an emulated qemu controller, 3 be an emulated PVUSB controller, and so on. == Open questions == Those are things I think I know; there are a couple of pertinent factual questions which I'm not sure of: * Is it possible to specify PVUSB controllers and attach USB devices to them before the guest is up and running (i.e., before the frontend is connected)? It looks like xend had a syntax for specifying virtual controllers and attaching virtual devices, so it seems like it should be possible. * Is it possible to connect a USB 1.1 device to a PVUSB controller which has been specified 2.0, or would there have to be a separate virtual controller for each? * Is it possible for the toolstack to tell if dom0 (or whatever the specified backend domain) has PVUSB support before starting the guest? * Is it possible for the toolstack to tell if domU has a working and connected PVUSB front-end? * Do we want to be able to create virtual hubs for qemu-backed controllers at some point in the future? Is there any groundwork we want to lay for that? == Design questions == Then based on that, we have several design questions. * How much of the "controller" thing should be exposed via libxl? Via xl? * This series only allows you to specify the "protocol", either PV or DEVICE_MODEL. Would it be better, for instance, to get rid of that, and instead allow the user to specify the creation of USB controllers, allow them to have a type of "HCI (or emulated)" or "PV", and allow the user to specify which controller to attach a specific device to? * How much "smarts" should we have in the libxl / xl about creating emulated/virtual controllers and of what kinds? * How much / what kind of "smarts" should be in libxl / xl about choosing a controller to plug a device into? * What about config file syntax? Should we try to reuse / extend the current config syntax, or create a new one? Should the new one allow the specification of controllers? Should it perhaps *require* the specification of controllers? == Draft design proposal == I've given it some thought, and based on the below is a suggested design for people to have a go at. Basic idea: Specify named controllers. It's controllers that are PV or emulated, and may have a backend domain. When adding devices, specify which controller to attach it to. Allow most things to have intelligent defaults. * libxl functions usb_controller_add usb_controller_remove/destroy usb_controller_list device struct: name # If empty, default to hciN/pvN, where N starts at 0 type = {PV, EMULATED, AUTO} backend_{domname,domid} # PV only usbversion = {1, 2, 3} numports # default 16 If type==AUTO, then it will be PV for a PV domain, EMULATED for an HVM domain. usb_add usb_remove/destroy usb_list device struct: controller_name # If empty, choose one of the controllers. type hostdev hostbus,hostaddr Note: I've removed backend from the device struct, as that will be based on the controller. * Storing information about virtual USB controllers / devices qemu does not at the moment have a way to query for plugged-in devices. I'm not sure if PVUSB does. Store information about both USB controllers and USB devices created with libxl in xenstore, somewhat similar to the system in the attached patches. * Domain creation I think what we add to the domain creation libxl calls will be obvious from how we design the config file interface. For reference, here are some example config snippets from the current xl / xm config files: -- snip -- # HVM USB usb=1 usbdevice=['tablet','host:4.3'] # HVM USB, not compatible with the above usbversion=3 # xend's PVUSB config file; this creates one virtual controller, then # plugs hostdev 1.8 into port 1 vusb=['usbver=2, numports=4, port_1=1-8'] -- snip -- Given that, here is a potential config file format: -- snip -- # Create two controllers, one pv, one emulated usbcontroller=['type=pv,name=pv0,usbversion=2,numports=4', 'type=emul,name=hci0,usbversion=2'] # Create a controller with the defaults; PV for PV guests, emul for HVM guests usbcontroller=[''] # Same as above, but defaulting to version 2 usbversion=2 # I think we should be able to automatically detect which format to # use; so I think we should re-use usbdevice. I could be convinced otherwise. usbdevice=['type=tablet','type=hostdev,hostaddr=4.3,bus=pv0'] -- snip -- Other ideas: * To make the interface closer to libvirt's, instead of specifying PV / EMULATED and usbversion, just specify a model. Then create "pvusb-v1", "pvusb-v2", &c for PVUSB hubs with the various versions, and detect automatically whether to use the PV or the qemu path. (NB that at the libvirt level, to fit with their syntax, we'd probably end up creating a "xenpvusbN" model in libvirt that would translate down to whatever the appropriate thing is in libxl.) * Rather than having to specify a controller, automatically hot-plug controllers as-needed. * Include an optional port number into which we will plug the USB device. Thoughts? -George == References == Last time this was posted: http://thread.gmane.org/gmane.comp.emulators.xen.devel/156796 PVUSB references: http://doc.opensuse.org/products/draft/SLES/SLES-xen_sd_draft/cha.xen.con... Libvirt references: http://wiki.libvirt.org/page/QEMUSwitchToLibvirt#-usbdevice https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Lin... http://libvirt.org/formatdomain.html http://libvirt.org/formatdomain.html#elementsControllers CC: Ian Jackson <ian.jackson(a)citrix.com&gt; CC: Roger Pau Monne <roger.pau(a)citrix.com&gt; CC: sstanisi(a)cbnco.com CC: Fabio Fantoni <fabio.fantoni(a)m2r.biz&gt; CC: Ian Campbell <ian.campbell(a)citrix.com&gt; CC: Anthony Perard <anthony.perard(a)citrix.com&gt; CC: Simon (Bo) Cao <caobosimon(a)gmail.com&gt; CC: LibVirt Development List <libvir-list(a)redhat.com&gt; CC: Jim Fehlig <jfehlig(a)suse.com&gt; CC: Daniel P. Berrange <berrange(a)redhat.com&gt;