Re: [RFC PATCH 0/5] qemu: Implement support for iommufd and multiple vSMMUs

Hi Daniel, On 5/20/2025 5:51 AM, Daniel P. Berrangé wrote: > > Hi, > > > > This is a follow up to the first RFC patchset [0] for supporting multiple > > vSMMU instances in a qemu VM. This patchset also introduces support for > > using iommufd to propagate DMA mappings to kernel for assigned devices. > > > > This patchset implements support for specifying multiple <iommu> devices > > within the VM definition when smmuv3Dev IOMMU model is specified, and is > > tested with Shameer's latest qemu RFC for HW-accelerated vSMMU devices [1] > > > > Moreover, it adds a new 'iommufd' member for virDomainIOMMUDef, > > in order to represent the iommufd object in qemu command line. This > > patchset also implements new 'iommufdId' and 'iommufdFd' attributes for > > hostdev devices to be associated with the iommufd object. > > > > For instance, specifying the iommufd object and associated hostdev in a > > VM definition with multiple IOMMUs, configured to be routed to > > pcie-expander-bus controllers in a way where VFIO device to SMMUv3 > > associations are matched with the host (pcie-expander-bus and > > pcie-root-port controllers are no longer auto-added/auto-routed > > like in the first revision of this RFC, as the PCIe topology will be > > configured by management apps): > > > > <devices> > > ... > > <controller type='pci' index='1' model='pcie-expander-bus'> > > <model name='pxb-pcie'/> > > <target busNr='252'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> > > </controller> > > <controller type='pci' index='2' model='pcie-expander-bus'> > > <model name='pxb-pcie'/> > > <target busNr='248'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> > > </controller> > > ... > > <controller type='pci' index='21' model='pcie-root-port'> > > <model name='pcie-root-port'/> > > <target chassis='21' port='0x0'/> > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> > > </controller> > > <controller type='pci' index='22' model='pcie-root-port'> > > <model name='pcie-root-port'/> > > <target chassis='22' port='0xa8'/> > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> > > </controller> > > ... > > <hostdev mode='subsystem' type='pci' managed='no'> > > <source> > > <address domain='0x0009' bus='0x01' slot='0x00' function='0x0'/> > > </source> > > <iommufdId>iommufd0</iommufdId> > > <address type='pci' domain='0x0000' bus='0x15' slot='0x00' function='0x0'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='no'> > > <source> > > <address domain='0x0019' bus='0x01' slot='0x00' function='0x0'/> > > </source> > > <iommufdId>iommufd0</iommufdId> > > <address type='pci' domain='0x0000' bus='0x16' slot='0x00' function='0x0'/> > > </hostdev> > > <iommu model='smmuv3Dev'> > > <iommufd> > > <id>iommufd0</id> > > </iommufd> > > <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> > IIUC, you're using <address> here to reference the earlier <controller> > pcie-expander-bus. This is a bit wierd as it is making it look like the > smmuv3Dev itself has a PCI address, but this is just the PCI address > of the controller. > > The smmuv3dev also doesn't have an address on the pcie-expander-bus, > it is just an association IIUC. > > So from this pov, I think I'd be inclined to say we should just > reference the <controller> based on its index, using an attribute > > <iommu model='smmuv3dev' controller='2'/> > I see, I will revise this to reference the controller index instead. > > </iommu> > > <iommu model='smmuv3Dev'> > > <iommufd> > > <id>iommufd0</id> > > </iommufd> > > <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> > > </iommu> > > </devices> > > > > This would get translated to a qemu command line with the arguments below: > > > > -device '{"driver":"pxb-pcie","bus_nr":252,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \ > > -device '{"driver":"pxb-pcie","bus_nr":248,"id":"pci.2","bus":"pcie.0","addr":"0x2"}' \ > > -device '{"driver":"pcie-root-port","port":0,"chassis":21,"id":"pci.21","bus":"pci.1","addr":"0x0"}' \ > > -device '{"driver":"pcie-root-port","port":168,"chassis":22,"id":"pci.22","bus":"pci.2","addr":"0x0"}' \ > > -object '{"qom-type":"iommufd","id":"iommufd0"}' \ > > -device '{"driver":"arm-smmuv3-accel","bus":"pci.1"}' \ > > -device '{"driver":"arm-smmuv3-accel","bus":"pci.2"}' \ > > -device '{"driver":"vfio-pci","host":"0009:01:00.0","id":"hostdev0","iommufd":"iommufd0","bus":"pci.21","addr":"0x0"}' \ > > -device '{"driver":"vfio-pci","host":"0019:01:00.0","id":"hostdev1","iommufd":"iommufd0","bus":"pci.22","addr":"0x0"}' \ > The iommufd integration in the XML looks a bit wierd too - we have > four different elements all referencing 'iommufd0' but nothing > is defining this. The iommu references the iommufd0, but nothing > actually uses this on the arm-smuv3-accel command line. > > > I've not been paying much attention to iommufd in QEMU, but IIUC > it will apply to x86_64 too. So I'm wondering how iommufd integration > sound work in libvirt more broadly. > It is my understanding that we want to consider device classes for libvirt device representation in XML, so I intended to have users declare the iommufd definition as an attribute under the <iommu> stanza, which would>

-object '{"qom-type":"iommufd","id":"iommufd0"}' but since this series implements support for multiple <iommu> definitions, we specify iommufd0 for multiple <iommu> stanzas. For x86_64, we would just specify the iommufd attribute once under a single <iommu> stanza. Would you suggest we move iommufd out of the <iommu> definition instead, like the examples below?