Re: [libvirt] mdevctl: A shoestring mediated device management and persistence utility

On Tue, Jun 18, 2019 at 1:01 PM Cornelia Huck <cohuck(a)redhat.com&gt; wrote: > On Mon, 17 Jun 2019 11:05:17 -0600 > Alex Williamson <alex.williamson(a)redhat.com&gt; wrote: > > > On Mon, 17 Jun 2019 16:10:30 +0100 > > Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote: > > > > > On Mon, Jun 17, 2019 at 08:54:38AM -0600, Alex Williamson wrote: > > > > On Mon, 17 Jun 2019 15:00:00 +0100 > > > > Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote: > > > > > > > > > On Thu, May 23, 2019 at 05:20:01PM -0600, Alex Williamson wrote: > > > > > > > Hi, > > > > > > > > > > > > Currently mediated device management, much like SR-IOV VF > management, > > > > > > is largely left as an exercise for the user. This is an attempt > to > > > > > > provide something and see where it goes. I doubt we'll solve > > > > > > everyone's needs on the first pass, but maybe we'll solve enough > and > > > > > > provide helpers for the rest. Without further ado, I'll point > to what > > > > > > I have so far: > > > > > > > > > > > > https://github.com/awilliam/mdevctl > > > > > > > > > > > > This is inspired by driverctl, which is also a bash utility. > mdevctl > > > > > > uses udev and systemd to record and recreate mdev devices for > > > > > > persistence and provides a command line utility for querying, > listing, > > > > > > starting, stopping, adding, and removing mdev devices. > Currently, for > > > > > > better or worse, it considers anything created to be > persistent. I can > > > > > > imagine a global configuration option that might disable this and > > > > > > perhaps an autostart flag per mdev device, such that mdevctl > might > > > > > > simply "know" about some mdevs but not attempt to create them > > > > > > automatically. Clearly command line usage help, man pages, and > > > > > > packaging are lacking as well, release early, release often, > plus this > > > > > > is a discussion starter to see if perhaps this is sufficient to > meet > > > > > > some needs. > > > > > > > > > > I think from libvirt's POV, we would *not* want devices to be made > > > > > unconditionally persistent. We usually wish to expose a choice to > > > > > applications whether to have resources be transient or persistent. > > > > > > > > > > So from that POV, a global config option to turn off persistence > > > > > is not workable either. We would want control per-device, with > > > > > autostart control per device too. > > > > > > > > The code has progressed somewhat in the past 3+ weeks, we still > persist > > > > all devices, but the start-up mode can be selected per device or > with a > > > > global default mode. Devices configured with 'auto' start-up > > > > automatically while 'manual' devices are simply known and available > to > > > > be started. I imagine we could add a 'transient' mode where we purge > > > > the information about the device when it is removed or the next time > > > > the parent device is added. > > > > > > Having a pesistent config written out & then purged later is still > > > problematic. If the host crashes, nothing will purge the config file, > > > so it will become a persistent device. Also when listing devices we > > > want to be able to report whether it is persistent or transient. The > > > obvious way todo that is to simply look if a config file exists or > > > not. > > > > I was thinking that the config file would identify the device as > > transient, therefore if the system crashed we'd have the opportunity to > > purge those entries on the next boot as we're processing the entries > > for that parent device. Clearly it has yet to be implemented, but I > > expect there are some advantages to tracking devices via a transient > > config entry or else we're constantly re-discovering foreign mdevs. > > I think we need to reach consensus about the actual scope of the > mdevctl tool. > > Thanks Cornelia, my thoughts: - Is it supposed to be responsible for managing *all* mdev devices in > the system, or is it more supposed to be a convenience helper for > users/software wanting to manage mdevs? > The latter. If an operator (or some software) wants to create mdevs by not using mdevctl (and rather directly calling the sysfs), I think it's OK. That said, mdevs created by mdevctl would be supported by systemctl, while the others not but I think it's okay.

- Do we want mdevctl to manage config files for individual mdevs, or > are they supposed to be in a common format that can also be managed > by e.g. libvirt? > Unless I misunderstand, I think mdevctl just helps to create mdevs for being used by guests created either by libvirt or QEMU or even others. How a guest would allocate a mdev (ie. saying "I'll use this specific mdev UUID") is IMHO not something for mdevctl.

- Should mdevctl be a stand-alone tool, provide library functions, or > both? Related: should it keep any internal state that is not written > to disk? (I think that also plays into the transient vs. persistent > question.)

FWIW, I'd love using mdevctl for OpenStack (Nova) just at least for creating persisted mdevs (ie. mdevs that would be recreated after rebooting using systemctl). That's the real use case I need. Whether libvirt would internally support mdevctl would be nice but that's not really something Nova needs, so I leave others providing their own thoughts. My personal opinion is that mdevctl should be able to tolerate mdevs > being configured by other means, but probably should not try to impose > its own configuration if it detects that (unless explicitly asked to do > so). Not sure how feasible that goal is. > > That's what I misunderstand : in order to have a guest using a vGPU, you need to do two things : 1/ create the mdev 2/ allocate this created dev to a specific guest config Of course, we could imagine a way to have both steps to be done directly by libvirt, but from my opinion, mdevctl is really helping 1/ and not 2/.

> A well-defined config file format is probably a win, even if it only > ends up being used by mdevctl itself.