Re: [RFC PATCH 0/5] qemu: Route hostdevs to multiple nested SMMUs

> Hi Daniel, > >> Top level libvirt device representation in XML is based on the device >> *class*, not the specific device impl. Adding a <nestedSmmuv3> device >> type XML element in libvirt is totally inappropriate. Any configuration >> must be done beneath the <iommu> element. > > Would keeping track of PXB <=> host SMMU nodes be better represented with a > <nestedSmmuv3> PXB attribute like below, when the "nestedSmmuv3" IOMMU model > is specified? This method would be simplest IMO because we could omit > keeping track of the nestedSmmuv3 bus number in the virDomainDef struct > since its association with a PXB controller would already be baked-in. > > <devices> > ... > <controller type='pci' index='1' model='pcie-expander-bus'> > <model name='pxb-pcie'/> > <target busNr='254'/> > <nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x0'/> > </controller> > ... > <iommu model='nestedSmmuv3'/> > </devices> > > Or would it still be preferred to purely contain the host SMMU node names > and bus numbers within the <iommu> element? If so, the virDomainDef struct > is setup to only have a single virDomainIOMMUDef member, but we need to keep > track of multiple host SMMU node names and bus numbers. Would we setup > multiple virDomainIOMMUDef members? Or add "char** nestedSmmuv3" and "size_t > *nestedSmmuv3Bus" members to the virDomainIOMMUDef struct to keep track of > these multiple host SMMU node names and bus numbers? > > Or we could modify the device info member of virDomainIOMMUDef to be a > variable-length array of device info structs instead of the "size_t > *nestedSmmuv3Bus" member. But I'm not convinced this would be the cleanest > approach, and the qemu command line doesn't specify a slot and function to > arm-smmuv3-nested devices - it just specifies bus number to keep track of > which PXB is associated with each SMMU node. > > If we need a way to associate PXB to SMMU node, we have multiple possible > approaches, listed below in order of best to worst in my opinion: > > 1. Adding a <nestedSmmuv3> attribute for PXB controller. > 2. Having a single virDomainIOMMUDef struct for virDomainDef. Adding > variable-length array members to virDomainIOMMUDef for multiple SMMU node > names and bus numbers. > 3. Having a single virDomainIOMMUDef struct for virDomainDef. Adding a > variable-length array member to virDomainIOMMUDef for SMMU node names. > Changing the single virDomainDeviceInfo struct to a variable-length array of > virDomainDeviceInfo structs for multiple nested SMMU bus numbers. > 4. Supporting multiple virDomainIOMMUDef structs for virDomainDef. > > I would appreciate your thoughts on which method to go with. Ignoring the specific QEMU impl, can you give a general outline of the relationship between host SMMU(s), guest SMMU(s) and PCI controllers, especially the M:N values of the relations.

Also, if this is getting associated with the host SMMU in some way, does this have implications for live migration compatibility ?