Re: [PATCH 5/5] node_device: mdev vfio-ccw support
On Wed, 9 Sep 2020 14:40:33 +0200
Erik Skultety <eskultet(a)redhat.com> wrote:
On Wed, Sep 09, 2020 at 08:23:39AM +0200, Bjoern Walk wrote:
> Erik Skultety <eskultet(a)redhat.com> [2020-09-08, 06:01PM +0200]:
> > On Mon, Aug 24, 2020 at 01:59:15PM +0200, Bjoern Walk wrote:
> > > From: Boris Fiuczynski <fiuczy(a)linux.ibm.com>
> > > + case VIR_NODE_DEV_CAP_SYSTEM:
> > > + case VIR_NODE_DEV_CAP_USB_DEV:
> > > + case VIR_NODE_DEV_CAP_USB_INTERFACE:
> > > + case VIR_NODE_DEV_CAP_NET:
> > > + case VIR_NODE_DEV_CAP_SCSI_HOST:
> > > + case VIR_NODE_DEV_CAP_SCSI_TARGET:
> > > + case VIR_NODE_DEV_CAP_SCSI:
> > > + case VIR_NODE_DEV_CAP_STORAGE:
> > > + case VIR_NODE_DEV_CAP_FC_HOST:
> > > + case VIR_NODE_DEV_CAP_VPORTS:
> > > + case VIR_NODE_DEV_CAP_SCSI_GENERIC:
> > > + case VIR_NODE_DEV_CAP_DRM:
> > > + case VIR_NODE_DEV_CAP_MDEV_TYPES:
> > > + case VIR_NODE_DEV_CAP_MDEV:
> > > + case VIR_NODE_DEV_CAP_CCW_DEV:
> >
> > What about ccw mdevs? I would think that it's CCW end point device that
> > we would want to assign concurrently, but we're only "slicing"
the root channel
> > subsystem device. If you have and CSS mdev, are the CCW subchannels separate
> > for each VM or does it behave kind of like NPIV with its vHBAs? OR I'm
> > completely off here?
>
> You are correct. This is an unfortunate architectural decision on our
> platform. For vfio-ccw, we instatiate the mdevs on the CSS layer, which
> has a 1-to-1 relation to the underlying CCW device. That's why we need
> the information about the CSS devices in libvirt, to give the user a
> chance to easily find this relation when he wants to do passthrough with
> a DASD for example. It's a bit unintuitive and even silly for the
> end user, but it's architecturally correct and hence it was implemented
> like this in the vfio-ccw kernel driver. We don't even get the benefits
> of something like NPIV because for every subchannel there is only one
> CCW device (at least to my knowledge).
Having read both Connie's blog on channels and the I/O chapter in the free
architectural publication you linked in the other reply, I now have a very
vague understating of the architecture. One thing that still
puzzles me though is that if there's a 1:1 subchannel mapping between CSS and an
CCW device which then communicates via a control unit (which there may be more
than 1) with CSS over up to 8 channel paths (this at least to me resembles
NPIV), what does the mdev really solve, can't you assign one of the channel
paths of one of the several control units the device is connected to to
plug into the VM? Well, clearly not, I'm just being curious and trying to
understand the use case here.
I think it's easier to leave control units mostly out of the picture,
as they are not really relevant for the modelling here. And yes, there
is always exactly one subchannel for an I/O device.
Subchannels (mostly a concept describing how to interact with a device)
and channel paths (mostly corresponding to the way data flows to/from
the device) are probably best understood as different ways to describe
how devices are accessed. You don't really want to pass channel paths
(as they are shared between many devices); the subchannel is a more
natural unit to pass. The operating system also has way more control
over subchannels than over individual channel paths.
Regarding mdev, the idea here is less to be able to slice up a device
as you do with GPUs, but more to be able to add a vendor driver that
does the actual interesting work (channel program translations etc.)
As to why we use the subchannel instead of the device, that's mostly
following from how the low-level channel I/O layer in the Linux kernel
is organized: The default I/O subchannel driver does a lot of things
that a normal device driver should not need to deal with (path
grouping, verification, status accumulation, etc.), but which would get
into the way if you wanted to assign the device via vfio. Therefore,
vfio-ccw implements an alternate I/O subchannel driver.
HTH (at least a little bit).