Re: Add iommu device when VM configured with > 255 vcpus
On 30/05/2024 18:00, Dario Faggioli via Devel wrote:
On Thu, 2024-05-30 at 14:45 +0200, Igor Mammedov wrote:
> Usability of huge VMs (incl. large amount of vCPUs) heavily depends
> on used guest OS. What would work for one might not work for other.
> For general purpose OS adding IOMMU is typically necessary to make
> vCPUs over 254 usable, for Windows adjusting -smp might make a
> difference.
>
> On the other hand, special built guest with tailored QEMU config
> might
> work without IOMMU just fine (David was the one who patched KVM to
> that
> effect if I recall correctly).
>
So, as far as I know, there is no way for any OS with any configuration
to bring up more than 255 vCPUs, without a vIOMMU.
IIUIC, it's a matter of number of bits available in the I/O APIC IRQ
destination register. Like, with only that available, and it being only
8 bit wide, it's just not doable.
On VMs, there's alternatively a KVM PV op to bump the limit to 32k vCPUs i.e.
KVM_FEATURE_MSI_EXT_DEST_ID
(on qemu it's cpu feature name +kvm-msi-ext-dest-id)
Which uses the other 24-bits for that destination register (which on hardware
would cross a page boundary in IOAPIC entry IIUC) without needing IOMMU
interrupt remapping. But you need the guest to understand that feature (which is
there since Linux v5.15 or around that timeframe). I think this is what Igor is
referring to.
Joao