Re: [PATCH 1/4] conf: add loader type 'none'

On Tue, Mar 28, 2023 at 04:46:45PM -0300, Daniel Henrique Barboza wrote: > On 3/22/23 13:37, Andrea Bolognani wrote: >> On Wed, Mar 22, 2023 at 10:36:20AM -0300, Daniel Henrique Barboza wrote: >>> I'm not sure if the OS overwrites the firmware when running bare metal. Usually >>> they provide different OS images for QEMU/libvirt and bare metal systems, probably >>> to account for these differences. Baking the firmware in the kernel like Fedora >>> Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading >>> the firmware by default, but now it's getting in the way. >>> >>> All this said, having a look at the recently updated Fedora for RISC-V wiki, the >>> instructions for booting with libvirt and QEMU are different. libvirt [1] is using >>> the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but >>> QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37. >> >> The libvirt instructions in that page seem to have been updated in >> the Fedora 30 timeframe, whereas the QEMU instructions appear to be >> current as of Fedora 37. >> >>> At first I don't see any particular reason of why this Fedora 37 image would work >>> on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt >>> and Fedora 37. If it works then this series becomes way less important. >> >> The libvirt instructions tell you to use >> >> --boot kernel=Fedora-Developer-Rawhide-*-fw_payload-uboot-qemu-virt-smode.elf >> --qemu-commandline='-bios none' >> >> (aka -bios none -kernel foo) while the QEMU ones suggest >> >> -bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin >> -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 >> >> Those are completely different approaches to booting the guest OS. >> The latter is much saner IMO, because it keeps the firmware under >> control of the host (as is the case for SeaBIOS/edk2) instead of >> mixing the kernel and the firmware and requiring guest-specific files >> to be extracted from the disk image before being able to boot. >> >> libvirt already provides the ability to pass arbitrary paths to -bios >> via the <loader type='rom'> element, so making the first part work >> should just be a matter of pointing it to the correct path. We don't >> seem to have the -device loader part wired up though, so that would >> need to be added. Probably as an additional attribute, in the vein of >> nvram.template? The address would have to be specified as well. >> >> Note that the firmware descriptor format already supports u-boot as >> the firmware type. So in the long run ideally you'd only need to >> specify >> >> <os firmware='uboot'> >> >> and, assuming the uboot-images-riscv64 package is installed on the >> host, everything should just work. > > So, I got the chance to try this out. We can remove the '-loader' from the > Fedora 37 command line. This: > > -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 > > Is loading the u-boot image in the address that the 'virt' machine loads the kernel. > So in the end it's similar to: > > -kernel u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb Excellent insight! I've confirmed that you are correct, and that <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000'/> </qemu:commandline> can indeed be replaced with <kernel>u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb</kernel> without apparently affecting the VM's ability to boot. > This means that RISC-V Fedora 37 is bootable without any libvirt changes. Similar to > what we already do with Ubuntu. I've spent some time on this as well. Here are my findings. First, let's briefly mention how RISC-V VMs had to be booted a few years back. This is described in detail at [1] and preceding sections, but the quick version is that you would have a single file containing (I believe) OpenSBI, u-boot and the kernel. This file would need to be extracted from the disk image, and would then be passed to -kernel. Having the firmware and the kernel in a single file, which lives on the host rather than the guest, was obviously extremely cumbersome and unsuitable for scenarios beyond experimentation. Today, things are looking a lot better: the firmware lives in the host and the kernel lives in the guest, as they should. However, there seem to be two different approaches to loading the firmware floating around. The first one, described in the Fedora wiki[2], involves passing two separate files to QEMU. They both come from the uboot-images-riscv64 package, and their use looks like -bios /usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin -kernel /usr/share/uboot/qemu-riscv64_spl/u-boot.itb As far as I understand, this is a u-boot binary, just divided in two parts and loaded separately. They might run in different hardware privilege contexts? I think the first file might also embed OpenSBI somehow. The second approach is the one described in the Ubuntu wiki[3], and also requires passing two files to QEMU, except this time they come from the opensbi and u-boot-qemu packages respectively. The usage looks like -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf I think in this case the first file is a minimal build of OpenSBI that likely just initializes enough hardware before handing control to an arbitrary payload - in this case, u-boot. There's a twist, however: the Ubuntu wiki claims that the -bios part is not necessary with QEMU >= 7.0, and I've verified that at least with the QEMU 7.2 package in Fedora 37 this is accurate. I believe this is because opensbi-riscv64-generic-fw_dynamic.bin, which is included in the QEMU package, gets loaded automatically, and this build of OpenSBI apparently also knows how to jump ahead to where -kernel loads u-boot.

I've tried mixing things, that is, boot the Fedora VM using the firmware taken from Ubuntu and vice versa, and thankfully it looks like this is perfectly fine. In conclusion, the current version of libvirt is perfectly capable of booting modern RISC-V guests through a combination of <loader> and <kernel> elements. That said, even if things nominally work, way too much handholding is necessary. So I'm wondering how we can push things forward to make the process more user-friendly. The obvious first step would be for the u-boot-qemu package in Ubuntu to ship a JSON descriptor file containing something like { "interface-types": [ "u-boot" ], "mapping": { "device": "kernel", "filename": "/usr/lib/u-boot/qemu-riscv64_smode/uboot.elf" }, "targets": [ { "architecture": "riscv64", "machines": [ "virt" ] } ] }

With some relatively straightforward plumbing on the libvirt side, we should be able to use the information provided in that file to make the simple VM configuration <os firmware='u-boot'> work out of the box. Getting the Fedora version to work would be trickier. As far as I can tell, there's currently no spec-compliant way to describe a firmware that requires both -bios and -kernel to be used at the same time. Should the spec be extended? Can we get Fedora to standardize on the, at least to an outside observer, simpler approach that Ubuntu has adopted?

[1] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_with_... [2] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_under... [3] https://wiki.ubuntu.com/RISC-V#Booting_with_QEMU