Re: Add iommu device when VM configured with > 255 vcpus
On Wed, 29 May 2024 14:44:52 -0400
Sergio Durigan Junior <sergio.durigan(a)canonical.com> wrote:
On Tuesday, May 28 2024, Jim Fehlig via Devel wrote:
> Hi All,
>
> I vaguely recall a discussion about $subject, but can't find it
> now. Perhaps buried in another thread. The topic has been raised
> internally again, and I'd like to gauge the community's interest in
> automatically adding the necessary devices/config when user has
> specified vcpus > 255.
>
> The comparison for prior art is a bit of a stretch, but we e.g. add
> <audio type='spice'/> when spice graphics is configured. I know
> libvirt has generally tried to avoid policy decisions, but it's not
> clear to me where we stand with cases such as this, where every x86 VM
> with > 255 vcpus needs a similarly configured iommu.
My two cents here: this is something I would certainly appreciate as a
downstream maintainer of QEMU/libvirt. In fact, I spent part of last
year figuring out and documenting the necessary bits that need to be put
together in order to use more than 288 vCPUs. One of the results of
this effort (with help from David Woodhouse) was:
https://ubuntu.com/server/docs/create-qemu-vms-with-up-to-1024-vcpus
I still have to write the equivalent guide for libvirt, FWIW.
Usability of huge VMs (incl. large amount of vCPUs) heavily depends
on used guest OS. What would work for one might not work for other.
For general purpose OS adding IOMMU is typically necessary to make
vCPUs over 254 usable, for Windows adjusting -smp might make a difference.
On the other hand, special built guest with tailored QEMU config might
work without IOMMU just fine (David was the one who patched KVM to that
effect if I recall correctly).
What I'm saying is that workload specific customization (guest OS)
is not handled at libvirt layer.
It's typically upper layers which know what guest OS would be
and it's up to them to customize config to make sure that workload
would be able to use VM as intended.
PS:
that said, having some guide in libvirt docs for a generic OS
(even just a linux) could be useful for folks working directly
with libvirt.
Cheers,