Re: [libvirt] [RFC PATCH 4/9] vfio-pci: register default dynamic-trap-bar-info region

On Fri, Dec 06, 2019 at 07:55:30AM +0800, Alex Williamson wrote: > On Wed, 4 Dec 2019 22:26:50 -0500 > Yan Zhao <yan.y.zhao(a)intel.com&gt; wrote: > > > Dynamic trap bar info region is a channel for QEMU and vendor driver to > > communicate dynamic trap info. It is of type > > VFIO_REGION_TYPE_DYNAMIC_TRAP_BAR_INFO and subtype > > VFIO_REGION_SUBTYPE_DYNAMIC_TRAP_BAR_INFO. > > > > This region has two fields: dt_fd and trap. > > When QEMU detects a device regions of this type, it will create an > > eventfd and write its eventfd id to dt_fd field. > > When vendor drivre signals this eventfd, QEMU reads trap field of this > > info region. > > - If trap is true, QEMU would search the device's PCI BAR > > regions and disable all the sparse mmaped subregions (if the sparse > > mmaped subregion is disablable). > > - If trap is false, QEMU would re-enable those subregions. > > > > A typical usage is > > 1. vendor driver first cuts its bar 0 into several sections, all in a > > sparse mmap array. So initally, all its bar 0 are passthroughed. > > 2. vendor driver specifys part of bar 0 sections to be disablable. > > 3. on migration starts, vendor driver signals dt_fd and set trap to true > > to notify QEMU disabling the bar 0 sections of disablable flags on. > > 4. QEMU disables those bar 0 section and hence let vendor driver be able > > to trap access of bar 0 registers and make dirty page tracking possible. > > 5. on migration failure, vendor driver signals dt_fd to QEMU again. > > QEMU reads trap field of this info region which is false and QEMU > > re-passthrough the whole bar 0 region. > > > > Vendor driver specifies whether it supports dynamic-trap-bar-info region > > through cap VFIO_PCI_DEVICE_CAP_DYNAMIC_TRAP_BAR in > > vfio_pci_mediate_ops->open(). > > > > If vfio-pci detects this cap, it will create a default > > dynamic_trap_bar_info region on behalf of vendor driver with region len=0 > > and region->ops=null. > > Vvendor driver should override this region's len, flags, rw, mmap in its > > vfio_pci_mediate_ops. > > TBH, I don't like this interface at all. Userspace doesn't pass data > to the kernel via INFO ioctls. We have a SET_IRQS ioctl for > configuring user signaling with eventfds. I think we only need to > define an IRQ type that tells the user to re-evaluate the sparse mmap > information for a region. The user would enumerate the device IRQs via > GET_IRQ_INFO, find one of this type where the IRQ info would also > indicate which region(s) should be re-evaluated on signaling. The user > would enable that signaling via SET_IRQS and simply re-evaluate the ok. I'll try to switch to this way. Thanks for this suggestion. > sparse mmap capability for the associated regions when signaled. Do you like the "disablable" flag of sparse mmap ? I think it's a lightweight way for user to switch mmap state of a whole region, otherwise going through a complete flow of GET_REGION_INFO and re-setup region might be too heavy.