Re: [libvirt] [edk2-discuss] [OVMF] resource assignment fails for passthrough PCI GPU

* Laszlo Ersek (lersek(a)redhat.com) wrote: > (+Dave, +Eduardo) > > On 11/22/19 00:00, dann frazier wrote: > > On Tue, Nov 19, 2019 at 06:06:15AM +0100, Laszlo Ersek wrote: > >> On 11/19/19 01:54, dann frazier wrote: > >>> On Fri, Nov 15, 2019 at 11:51:18PM +0100, Laszlo Ersek wrote: > >>>> On 11/15/19 19:56, dann frazier wrote: > >>>>> Hi, > >>>>> I'm trying to passthrough an Nvidia GPU to a q35 KVM guest, but UEFI > >>>>> is failing to allocate resources for it. I have no issues if I boot w/ > >>>>> a legacy BIOS, and it works fine if I tell the linux guest to do the > >>>>> allocation itself - but I'm looking for a way to make this work w/ > >>>>> OVMF by default. > >>>>> > >>>>> I posted a debug log here: > >>>>> https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563/+attachment/5... > >>>>> > >>>>> Linux guest lspci output is also available for both seabios/OVMF boots here: > >>>>> https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1849563 > >>>> > >>>> By default, OVMF exposes such a 64-bit MMIO aperture for PCI MMIO BAR > >>>> allocation that is 32GB in size. The generic PciBusDxe driver collects, > >>>> orders, and assigns / allocates the MMIO BARs, but it can work only out > >>>> of the aperture that platform code advertizes. > >>>> > >>>> Your GPU's region 1 is itself 32GB in size. Given that there are further > >>>> PCI devices in the system with further 64-bit MMIO BARs, the default > >>>> aperture cannot accommodate everything. In such an event, PciBusDxe > >>>> avoids assigning the largest BARs (to my knowledge), in order to > >>>> conserve the most aperture possible, for other devices -- hence break > >>>> the fewest possible PCI devices. > >>>> > >>>> You can control the aperture size from the QEMU command line. You can > >>>> also do it from the libvirt domain XML, technically speaking. The knob > >>>> is experimental, so no stability or compatibility guarantees are made. > >>>> (That's also the reason why it's a bit of a hack in the libvirt domain XML.) > >>>> > >>>> The QEMU cmdline options is described in the following edk2 commit message: > >>>> > >>>> https://github.com/tianocore/edk2/commit/7e5b1b670c38 > >>> > >>> Hi Laszlo, > >>> > >>> Thanks for taking the time to describe this in detail! The -fw_cfg > >>> option did avoid the problem for me. > >> > >> Good to hear, thanks. > >> > >>> I also noticed that the above > >>> commit message mentions the existence of a 24GB card as a reasoning > >>> behind choosing the 32GB default aperture. From what you say below, I > >>> understand that bumping this above 64GB could break hosts w/ <= 37 > >>> physical address bits. > >> > >> Right. > >> > >>> What would be the downside of bumping the > >>> default aperture to, say, 48GB? > >> > >> The placement of the aperture is not trivial (please see the code > >> comments in the linked commit). The base address of the aperture is > >> chosen so that the largest BAR that can fit in the aperture may be > >> naturally aligned. (BARs are whole powers of two.) > >> > >> The largest BAR that can fit in a 48 GB aperture is 32 GB. Therefore > >> such an aperture would be aligned at 32 GB -- the lowest base address > >> (dependent on guest RAM size) would be 32 GB. Meaning that the aperture > >> would end at 32 + 48 = 80 GB. That still breaches the 36-bit phys > >> address width. > >> > >> 32 GB is the largest aperture size that can work with 36-bit phys > >> address width; that's the aperture that ends at 64 GB exactly. > > > > Thanks, yeah - now that I read the code comments that is clear (as > > clear as it can be w/ my low level of base knowledge). In the commit you > > mention Gerd (CC'd) had suggested a heuristic-based approach for > > sizing the aperture. When you say "PCPU address width" - is that a > > function of the available physical bits? > > "PCPU address width" is not a "function" of the available physical bits > -- it *is* the available physical bits. "PCPU" simply stands for > "physical CPU". > > > IOW, would that approach > > allow OVMF to automatically grow the aperture to the max ^2 supported > > by the host CPU? > > Maybe. > > The current logic in OVMF works from the guest-physical address space > size -- as deduced from multiple factors, such as the 64-bit MMIO > aperture size, and others -- towards the guest-CPU (aka VCPU) address > width. The VCPU address width is important for a bunch of other purposes > in the firmware, so OVMF has to calculate it no matter what. > > Again, the current logic is to calculate the highest guest-physical > address, and then deduce the VCPU address width from that (and then > expose it to the rest of the firmware). > > Your suggestion would require passing the PCPU (physical CPU) address > width from QEMU/KVM into the guest, and reversing the direction of the > calculation. The PCPU address width would determine the VCPU address > width directly, and then the 64-bit PCI MMIO aperture would be > calculated from that. > > However, there are two caveats. > > (1) The larger your guest-phys address space (as exposed through the > VCPU address width to the rest of the firmware), the more guest RAM you > need for page tables. Because, just before entering the DXE phase, the > firmware builds 1:1 mapping page tables for the entire guest-phys > address space. This is necessary e.g. so you can access any PCI MMIO BAR. > > Now consider that you have a huge beefy virtualization host with say 46 > phys address bits, and a wimpy guest with say 1.5GB of guest RAM. Do you > absolutely want tens of *terabytes* for your 64-bit PCI MMIO aperture? > Do you really want to pay for the necessary page tables with that meager > guest RAM? > > (Such machines do exist BTW, for example: > > http://mid.mail-archive.com/9BD73EA91F8E404F851CF3F519B14AA8036C67B5@DGGE... > ) > > In other words, you'd need some kind of knob anyway, because otherwise > your aperture could grow too *large*. > > > (2) Exposing the PCPU address width to the guest may have nasty > consequences at the QEMU/KVM level, regardless of guest firmware. For > example, that kind of "guest enlightenment" could interfere with migration. > > If you boot a guest let's say with 16GB of RAM, and tell it "hey friend, > have 40 bits of phys address width!", then you'll have a difficult time > migrating that guest to a host with a CPU that only has 36-bits wide > physical addresses -- even if the destination host has plenty of RAM > otherwise, such as a full 64GB. > > There could be other QEMU/KVM / libvirt issues that I m unaware of > (hence the CC to Dave and Eduardo). host physical address width gets messy. There are differences as well between upstream qemu behaviour, and some downstreams. I think the story is that: a) Qemu default: 40 bits on any host b) -cpu blah,host-phys-bits=true to follow the host. c) RHEL has host-phys-bits=true by default As you say, the only real problem with host-phys-bits is migration - between say an E3 and an E5 xeon with different widths. The magic 40's is generally wrong as well - I think it came from some ancient AMD, but it's the default on QEMU TCG as well.

I don't think there's a way to set it in libvirt; https://bugzilla.redhat.com/show_bug.cgi?id=1578278 is a bz asking for that. IMHO host-phys-bits is actually pretty safe; and makes most sense in a lot of cases.

Dave > Thanks, > Laszlo > > > > > -dann > > > >>>> For example, to set a 64GB aperture, pass: > >>>> > >>>> -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536 > >>>> > >>>> The libvirt domain XML syntax is a bit tricky (and it might "taint" your > >>>> domain, as it goes outside of the QEMU features that libvirt directly > >>>> maps to): > >>>> > >>>> <domain > >>>> type='kvm' > >>>> xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> > >>>> <qemu:commandline> > >>>> <qemu:arg value='-fw_cfg'/> > >>>> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> > >>>> </qemu:commandline> > >>>> </domain> > >>>> > >>>> Some notes: > >>>> > >>>> (1) The "xmlns:qemu" namespace definition attribute in the <domain> root > >>>> element is important. You have to add it manually when you add > >>>> <qemu:commandline> and <qemu:arg> too. Without the namespace > >>>> definition, the latter elements will make no sense, and libvirt will > >>>> delete them immediately. > >>>> > >>>> (2) The above change will grow your guest's physical address space to > >>>> more than 64GB. As a consequence, on your *host*, *if* your physical CPU > >>>> supports nested paging (called "ept" on Intel and "npt" on AMD), *then* > >>>> the CPU will have to support at least 37 physical address bits too, for > >>>> the guest to work. Otherwise, the guest will break, hard. > >>>> > >>>> Here's how to verify (on the host): > >>>> > >>>> (2a) run "egrep -w 'npt|ept' /proc/cpuinfo" --> if this does not produce > >>>> output, then stop reading here; things should work. Your CPU does not > >>>> support nested paging, so KVM will use shadow paging, which is slower, > >>>> but at least you don't have to care about the CPU's phys address width. > >>>> > >>>> (2b) otherwise (i.e. when you do have nested paging), run "grep 'bits > >>>> physical' /proc/cpuinfo" --> if the physical address width is >=37, > >>>> you're good. > >>>> > >>>> (2c) if you have nested paging but exactly 36 phys address bits, then > >>>> you'll have to forcibly disable nested paging (assuming you want to run > >>>> a guest with larger than 64GB guest-phys address space, that is). On > >>>> Intel, issue: > >>>> > >>>> rmmod kvm_intel > >>>> modprobe kvm_intel ept=N > >>>> > >>>> On AMD, go with: > >>>> > >>>> rmmod kvm_amd > >>>> modprobe kvm_amd npt=N > >>>> > >>>> Hope this helps, > >>>> Laszlo > >>>> > >>> > >> > > > -- Dr. David Alan Gilbert / dgilbert(a)redhat.com / Manchester, UK