On 5/30/2025 8:05 AM, Daniel P. Berrangé wrote:
Hi Daniel,
On 5/20/2025 5:51 AM, Daniel P. Berrangé wrote:
Hi,
This is a follow up to the first RFC patchset [0] for supporting multiple vSMMU instances in a qemu VM. This patchset also introduces support for using iommufd to propagate DMA mappings to kernel for assigned devices.
This patchset implements support for specifying multiple <iommu> devices within the VM definition when smmuv3Dev IOMMU model is specified, and is tested with Shameer's latest qemu RFC for HW-accelerated vSMMU devices [1]
Moreover, it adds a new 'iommufd' member for virDomainIOMMUDef, in order to represent the iommufd object in qemu command line. This patchset also implements new 'iommufdId' and 'iommufdFd' attributes for hostdev devices to be associated with the iommufd object.
For instance, specifying the iommufd object and associated hostdev in a VM definition with multiple IOMMUs, configured to be routed to pcie-expander-bus controllers in a way where VFIO device to SMMUv3 associations are matched with the host (pcie-expander-bus and pcie-root-port controllers are no longer auto-added/auto-routed like in the first revision of this RFC, as the PCIe topology will be configured by management apps):
<devices> ... <controller type='pci' index='1' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='252'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='248'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> ... <controller type='pci' index='21' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='21' port='0x0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='22' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='22' port='0xa8'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> ... <hostdev mode='subsystem' type='pci' managed='no'> <source> <address domain='0x0009' bus='0x01' slot='0x00' function='0x0'/> </source> <iommufdId>iommufd0</iommufdId> <address type='pci' domain='0x0000' bus='0x15' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='no'> <source> <address domain='0x0019' bus='0x01' slot='0x00' function='0x0'/> </source> <iommufdId>iommufd0</iommufdId> <address type='pci' domain='0x0000' bus='0x16' slot='0x00' function='0x0'/> </hostdev> <iommu model='smmuv3Dev'> <iommufd> <id>iommufd0</id> </iommufd> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> IIUC, you're using <address> here to reference the earlier <controller> pcie-expander-bus. This is a bit wierd as it is making it look like the smmuv3Dev itself has a PCI address, but this is just the PCI address of the controller.
The smmuv3dev also doesn't have an address on the pcie-expander-bus, it is just an association IIUC.
So from this pov, I think I'd be inclined to say we should just reference the <controller> based on its index, using an attribute
<iommu model='smmuv3dev' controller='2'/>
I see, I will revise this to reference the controller index instead.
</iommu> <iommu model='smmuv3Dev'> <iommufd> <id>iommufd0</id> </iommufd> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </iommu> </devices>
This would get translated to a qemu command line with the arguments below:
-device '{"driver":"pxb-pcie","bus_nr":252,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \ -device '{"driver":"pxb-pcie","bus_nr":248,"id":"pci.2","bus":"pcie.0","addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":0,"chassis":21,"id":"pci.21","bus":"pci.1","addr":"0x0"}' \ -device '{"driver":"pcie-root-port","port":168,"chassis":22,"id":"pci.22","bus":"pci.2","addr":"0x0"}' \ -object '{"qom-type":"iommufd","id":"iommufd0"}' \ -device '{"driver":"arm-smmuv3-accel","bus":"pci.1"}' \ -device '{"driver":"arm-smmuv3-accel","bus":"pci.2"}' \ -device '{"driver":"vfio-pci","host":"0009:01:00.0","id":"hostdev0","iommufd":"iommufd0","bus":"pci.21","addr":"0x0"}' \ -device '{"driver":"vfio-pci","host":"0019:01:00.0","id":"hostdev1","iommufd":"iommufd0","bus":"pci.22","addr":"0x0"}' \
The iommufd integration in the XML looks a bit wierd too - we have four different elements all referencing 'iommufd0' but nothing is defining this. The iommu references the iommufd0, but nothing actually uses this on the arm-smuv3-accel command line.
I've not been paying much attention to iommufd in QEMU, but IIUC it will apply to x86_64 too. So I'm wondering how iommufd integration sound work in libvirt more broadly.
It is my understanding that we want to consider device classes for libvirt device representation in XML, so I intended to have users declare the iommufd definition as an attribute under the <iommu> stanza, which would> translate to the following qemu argument: -object '{"qom-type":"iommufd","id":"iommufd0"}'
but since this series implements support for multiple <iommu> definitions, we specify iommufd0 for multiple <iommu> stanzas. For x86_64, we would just specify the iommufd attribute once under a single <iommu> stanza.
Would you suggest we move iommufd out of the <iommu> definition instead, like the examples below? AFAICT iommufd isn't connected to the guest iommu at all in terms of configuration, it is simply an attribute of the hostdev. eg we could do
<hostdev mode='subsystem' type='mdev' model='vfio-pci' iommufd='on'>
that does leave open the possibility that someone configures iommufd on one hostdev, but not on another, but that's not as bad as when we set it on the <iommu> too. So something we can validate in post-parse logic if we need to ensure consistent usage - if qemu allows a mix of iommfd and non-iommufd for vfio-pci, we can just allow that at libvirt too
Ok sounds good, I will make this only a hostdev attribute and generate the qemu iommufd object argument when detected instead of having users specify it under the guest iommu definition. We should be able to allow a mix of iommufd and non-iommufd for vfio-pci, and we can avoid the case where someone specifies it for the hostdev but not in the <iommu> stanza like you mentioned. Thanks, Nathan