Re: [RFC PATCH 0/5] qemu: Route hostdevs to multiple nested SMMUs

> Hi Daniel, > >> Top level libvirt device representation in XML is based on the device >> *class*, not the specific device impl. Adding a <nestedSmmuv3> device >> type XML element in libvirt is totally inappropriate. Any configuration >> must be done beneath the <iommu> element. > Would keeping track of PXB <=> host SMMU nodes be better represented with a > <nestedSmmuv3> PXB attribute like below, when the "nestedSmmuv3" IOMMU model > is specified? This method would be simplest IMO because we could omit > keeping track of the nestedSmmuv3 bus number in the virDomainDef struct > since its association with a PXB controller would already be baked-in. > > <devices> > ... > <controller type='pci' index='1' model='pcie-expander-bus'> > <model name='pxb-pcie'/> > <target busNr='254'/> > <nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> > </controller> > ... > <iommu model='nestedSmmuv3'/> > </devices> The '<nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3>' bit is also wierd, because it never appears in the QEMU command line at all. Some kind of implicit association seems to be taking place behind the scenes and that is unlikely to be a desirable thing. We need to model associations between devices explicitly and directly pass this on to QEMU. If the nestedSmmuv3 is assocaited with a host SMMUv3 device, then that association should be represented on the <iommu> device, and this mapping passed to QEMU.

> Or would it still be preferred to purely contain the host SMMU node names > and bus numbers within the <iommu> element? If so, the virDomainDef struct > is setup to only have a single virDomainIOMMUDef member, but we need to keep > track of multiple host SMMU node names and bus numbers. Would we setup > multiple virDomainIOMMUDef members? Or add "char** nestedSmmuv3" and "size_t > *nestedSmmuv3Bus" members to the virDomainIOMMUDef struct to keep track of > these multiple host SMMU node names and bus numbers? > > Or we could modify the device info member of virDomainIOMMUDef to be a > variable-length array of device info structs instead of the "size_t > *nestedSmmuv3Bus" member. But I'm not convinced this would be the cleanest > approach, and the qemu command line doesn't specify a slot and function to > arm-smmuv3-nested devices - it just specifies bus number to keep track of > which PXB is associated with each SMMU node. > > If we need a way to associate PXB to SMMU node, we have multiple possible > approaches, listed below in order of best to worst in my opinion: > > 1. Adding a <nestedSmmuv3> attribute for PXB controller. > 2. Having a single virDomainIOMMUDef struct for virDomainDef. Adding > variable-length array members to virDomainIOMMUDef for multiple SMMU node > names and bus numbers. > 3. Having a single virDomainIOMMUDef struct for virDomainDef. Adding a > variable-length array member to virDomainIOMMUDef for SMMU node names. > Changing the single virDomainDeviceInfo struct to a variable-length array of > virDomainDeviceInfo structs for multiple nested SMMU bus numbers. > 4. Supporting multiple virDomainIOMMUDef structs for virDomainDef. We need (4). Each device must be represented as a distinct virDomainIOMMUDef, and then associated to the controller and host SMMU.