-----Original Message----- From: Nathan Chen <nathanc@nvidia.com> Sent: Thursday, January 30, 2025 5:52 PM To: Daniel P. Berrangé <berrange@redhat.com> Cc: devel@lists.libvirt.org; Nicolin Chen <nicolinc@nvidia.com>; Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> Subject: Re: [RFC PATCH 0/5] qemu: Route hostdevs to multiple nested SMMUs
Hi Daniel,
Top level libvirt device representation in XML is based on the device *class*, not the specific device impl. Adding a <nestedSmmuv3> device type XML element in libvirt is totally inappropriate. Any configuration must be done beneath the <iommu> element. Would keeping track of PXB <=> host SMMU nodes be better represented with a <nestedSmmuv3> PXB attribute like below, when the "nestedSmmuv3" IOMMU model is specified? This method would be simplest IMO because we could omit keeping track of the nestedSmmuv3 bus number in the virDomainDef struct since its association with a PXB controller would already be baked-in.
<devices> ... <controller type='pci' index='1' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='254'/> <nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller> ... <iommu model='nestedSmmuv3'/> </devices>
The '<nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3>' bit is also wierd, because it never appears in the QEMU command line at all. Some kind of implicit association seems to be taking place behind the scenes and that is unlikely to be a desirable thing. We need to model associations between devices explicitly and directly pass this on to QEMU.
If the nestedSmmuv3 is assocaited with a host SMMUv3 device, then that association should be represented on the <iommu> device, and this mapping passed to QEMU.
The '<nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3>' bit will be moved to the <iommu> definition in the next revision.
I agree that explicitly specifying the host SMMU node name in QEMU would make the host to guest SMMU association more clear. But the current QEMU implementation appears more flexible in allowing us to create nestedSmmuv3 instances and then imply the association to a host SMMU node based on the PCIe topology isolating devices under different PXBs. I think we should discuss this further on the QEMU thread with Shameer before he sends out the second revision of his series, then adjust the libvirt implementation based on the next QEMU revision.
I just replied to Daniel's same comment on that thread .Sorry forgot to CC you. It's here, https://lore.kernel.org/qemu-devel/7ecabe74e0514367baf28d67675e5db8@huawei.c... Thanks, Shameer.