On Wed, Mar 22, 2023 at 09:32:21AM -0300, Daniel Henrique Barboza wrote: > On 3/22/23 06:42, Daniel Henrique Barboza wrote: >> On 3/22/23 06:25, Daniel P. Berrangé wrote: >>> On Wed, Mar 22, 2023 at 06:10:18AM -0300, Daniel Henrique Barboza wrote: >>>> +    if (loader->type != VIR_DOMAIN_LOADER_TYPE_NONE || >>>> +        (loader->type == VIR_DOMAIN_LOADER_TYPE_NONE && !loader->path)) >>>>           virBufferAsprintf(&loaderAttrBuf, " type='%s'", >>>>                             virDomainLoaderTypeToString(loader->type)); >>> >>> VIR_DOMAIN_LOADER_TYPE_NONE is a constant we're already using internally >>> to track when the user has not given any <loader> element at all. >>> >>> It is not a good idea to overload for this for the user explicitly >>> requesting a config. >> >> Makes sense. Any name suggestions for this new constant? >> >> VIR_DOMAIN_LOADER_TYPE_SKIP is the first thing that comes to mind. If we want to >> keep it consistent we should also change this new type to <loader type='skip'/> >> as well. > > VIR_DOMAIN_LOADER_TYPE_OTHER seems more appropriate to indicate that the firmware > will be loaded not by QEMU, but other means (e.g. kernel). I'm not sure we really need a new loader type. Can't we just add '-bios none' unconditionally on RISC-V when a loader path or a kernel path have been provided, either manually or via autoselection?

Today, trying to boot a RISC-V Fedora Rawhide image in a RISC-V QEMU domain results in the following error: ==== error: Failed to start domain 'riscv-fedora' error: internal error: process exited while connecting to monitor: 2023-03-20T17:31:02.650862Z qemu-system-riscv64: Some ROM regions are overlapping These ROM regions might have been loaded by direct user request or by default. They could be BIOS/firmware images, a guest kernel, initrd or some other file loaded into guest memory. Check whether you intended to load all this guest code, and whether it has been built to load to the correct addresses. ==== This happens because the default RISC-V QEMU firmware, OpenSBI, is always loaded unless "-bios none" is passed in the command line, and the Fedora Rawhide guest kernel has its own ROM. Other machines such as PPC64 'pseries' shows the same behavior: the default firmware is always loaded unless specified otherwise with the '-bios' option.

I'm not sure if the OS overwrites the firmware when running bare metal. Usually they provide different OS images for QEMU/libvirt and bare metal systems, probably to account for these differences. Baking the firmware in the kernel like Fedora Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading the firmware by default, but now it's getting in the way. All this said, having a look at the recently updated Fedora for RISC-V wiki, the instructions for booting with libvirt and QEMU are different. libvirt [1] is using the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37.

At first I don't see any particular reason of why this Fedora 37 image would work on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt and Fedora 37. If it works then this series becomes way less important.

On Wed, Mar 22, 2023 at 11:37:10AM -0500, Andrea Bolognani wrote: > Note that the firmware descriptor format already supports u-boot as > the firmware type. So in the long run ideally you'd only need to > specify > > <os firmware='uboot'> > > and, assuming the uboot-images-riscv64 package is installed on the > host, everything should just work. And that would bring alignment with other architectures approach, which is preferrable to adding a special hack needed for riscv, because the latter is something that most mgmt apps will forget to use.

Of course even better would be for riscv64 to use UEFI :-)

On 3/22/23 13:37, Andrea Bolognani wrote: > On Wed, Mar 22, 2023 at 10:36:20AM -0300, Daniel Henrique Barboza wrote: > > I'm not sure if the OS overwrites the firmware when running bare metal. Usually > > they provide different OS images for QEMU/libvirt and bare metal systems, probably > > to account for these differences. Baking the firmware in the kernel like Fedora > > Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading > > the firmware by default, but now it's getting in the way. > > > > All this said, having a look at the recently updated Fedora for RISC-V wiki, the > > instructions for booting with libvirt and QEMU are different. libvirt [1] is using > > the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but > > QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37. > > The libvirt instructions in that page seem to have been updated in > the Fedora 30 timeframe, whereas the QEMU instructions appear to be > current as of Fedora 37. > > > At first I don't see any particular reason of why this Fedora 37 image would work > > on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt > > and Fedora 37. If it works then this series becomes way less important. > > The libvirt instructions tell you to use > > --boot kernel=Fedora-Developer-Rawhide-*-fw_payload-uboot-qemu-virt-smode.elf > --qemu-commandline='-bios none' > > (aka -bios none -kernel foo) while the QEMU ones suggest > > -bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin > -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 > > Those are completely different approaches to booting the guest OS. > The latter is much saner IMO, because it keeps the firmware under > control of the host (as is the case for SeaBIOS/edk2) instead of > mixing the kernel and the firmware and requiring guest-specific files > to be extracted from the disk image before being able to boot. > > libvirt already provides the ability to pass arbitrary paths to -bios > via the <loader type='rom'> element, so making the first part work > should just be a matter of pointing it to the correct path. We don't > seem to have the -device loader part wired up though, so that would > need to be added. Probably as an additional attribute, in the vein of > nvram.template? The address would have to be specified as well. > > Note that the firmware descriptor format already supports u-boot as > the firmware type. So in the long run ideally you'd only need to > specify > > <os firmware='uboot'> > > and, assuming the uboot-images-riscv64 package is installed on the > host, everything should just work. So, I got the chance to try this out. We can remove the '-loader' from the Fedora 37 command line. This: -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 Is loading the u-boot image in the address that the 'virt' machine loads the kernel. So in the end it's similar to: -kernel u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb

This means that RISC-V Fedora 37 is bootable without any libvirt changes. Similar to what we already do with Ubuntu.

On Tue, Mar 28, 2023 at 04:46:45PM -0300, Daniel Henrique Barboza wrote: > On 3/22/23 13:37, Andrea Bolognani wrote: >> On Wed, Mar 22, 2023 at 10:36:20AM -0300, Daniel Henrique Barboza wrote: >>> I'm not sure if the OS overwrites the firmware when running bare metal. Usually >>> they provide different OS images for QEMU/libvirt and bare metal systems, probably >>> to account for these differences. Baking the firmware in the kernel like Fedora >>> Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading >>> the firmware by default, but now it's getting in the way. >>> >>> All this said, having a look at the recently updated Fedora for RISC-V wiki, the >>> instructions for booting with libvirt and QEMU are different. libvirt [1] is using >>> the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but >>> QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37. >> >> The libvirt instructions in that page seem to have been updated in >> the Fedora 30 timeframe, whereas the QEMU instructions appear to be >> current as of Fedora 37. >> >>> At first I don't see any particular reason of why this Fedora 37 image would work >>> on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt >>> and Fedora 37. If it works then this series becomes way less important. >> >> The libvirt instructions tell you to use >> >> --boot kernel=Fedora-Developer-Rawhide-*-fw_payload-uboot-qemu-virt-smode.elf >> --qemu-commandline='-bios none' >> >> (aka -bios none -kernel foo) while the QEMU ones suggest >> >> -bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin >> -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 >> >> Those are completely different approaches to booting the guest OS. >> The latter is much saner IMO, because it keeps the firmware under >> control of the host (as is the case for SeaBIOS/edk2) instead of >> mixing the kernel and the firmware and requiring guest-specific files >> to be extracted from the disk image before being able to boot. >> >> libvirt already provides the ability to pass arbitrary paths to -bios >> via the <loader type='rom'> element, so making the first part work >> should just be a matter of pointing it to the correct path. We don't >> seem to have the -device loader part wired up though, so that would >> need to be added. Probably as an additional attribute, in the vein of >> nvram.template? The address would have to be specified as well. >> >> Note that the firmware descriptor format already supports u-boot as >> the firmware type. So in the long run ideally you'd only need to >> specify >> >> <os firmware='uboot'> >> >> and, assuming the uboot-images-riscv64 package is installed on the >> host, everything should just work. > > So, I got the chance to try this out. We can remove the '-loader' from the > Fedora 37 command line. This: > > -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 > > Is loading the u-boot image in the address that the 'virt' machine loads the kernel. > So in the end it's similar to: > > -kernel u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb Excellent insight! I've confirmed that you are correct, and that <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000'/> </qemu:commandline> can indeed be replaced with <kernel>u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb</kernel> without apparently affecting the VM's ability to boot. > This means that RISC-V Fedora 37 is bootable without any libvirt changes. Similar to > what we already do with Ubuntu. I've spent some time on this as well. Here are my findings. First, let's briefly mention how RISC-V VMs had to be booted a few years back. This is described in detail at [1] and preceding sections, but the quick version is that you would have a single file containing (I believe) OpenSBI, u-boot and the kernel. This file would need to be extracted from the disk image, and would then be passed to -kernel. Having the firmware and the kernel in a single file, which lives on the host rather than the guest, was obviously extremely cumbersome and unsuitable for scenarios beyond experimentation. Today, things are looking a lot better: the firmware lives in the host and the kernel lives in the guest, as they should. However, there seem to be two different approaches to loading the firmware floating around. The first one, described in the Fedora wiki[2], involves passing two separate files to QEMU. They both come from the uboot-images-riscv64 package, and their use looks like -bios /usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin -kernel /usr/share/uboot/qemu-riscv64_spl/u-boot.itb As far as I understand, this is a u-boot binary, just divided in two parts and loaded separately. They might run in different hardware privilege contexts? I think the first file might also embed OpenSBI somehow. The second approach is the one described in the Ubuntu wiki[3], and also requires passing two files to QEMU, except this time they come from the opensbi and u-boot-qemu packages respectively. The usage looks like -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf I think in this case the first file is a minimal build of OpenSBI that likely just initializes enough hardware before handing control to an arbitrary payload - in this case, u-boot. There's a twist, however: the Ubuntu wiki claims that the -bios part is not necessary with QEMU >= 7.0, and I've verified that at least with the QEMU 7.2 package in Fedora 37 this is accurate. I believe this is because opensbi-riscv64-generic-fw_dynamic.bin, which is included in the QEMU package, gets loaded automatically, and this build of OpenSBI apparently also knows how to jump ahead to where -kernel loads u-boot.

I've tried mixing things, that is, boot the Fedora VM using the firmware taken from Ubuntu and vice versa, and thankfully it looks like this is perfectly fine. In conclusion, the current version of libvirt is perfectly capable of booting modern RISC-V guests through a combination of <loader> and <kernel> elements. That said, even if things nominally work, way too much handholding is necessary. So I'm wondering how we can push things forward to make the process more user-friendly. The obvious first step would be for the u-boot-qemu package in Ubuntu to ship a JSON descriptor file containing something like { "interface-types": [ "u-boot" ], "mapping": { "device": "kernel", "filename": "/usr/lib/u-boot/qemu-riscv64_smode/uboot.elf" }, "targets": [ { "architecture": "riscv64", "machines": [ "virt" ] } ] }

With some relatively straightforward plumbing on the libvirt side, we should be able to use the information provided in that file to make the simple VM configuration <os firmware='u-boot'> work out of the box. Getting the Fedora version to work would be trickier. As far as I can tell, there's currently no spec-compliant way to describe a firmware that requires both -bios and -kernel to be used at the same time. Should the spec be extended? Can we get Fedora to standardize on the, at least to an outside observer, simpler approach that Ubuntu has adopted?

[1] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_with_... [2] https://fedoraproject.org/wiki/Architectures/RISC-V/Installing#Boot_under... [3] https://wiki.ubuntu.com/RISC-V#Booting_with_QEMU

The second approach is the one described in the Ubuntu wiki[3], and also requires passing two files to QEMU, except this time they come from the opensbi and u-boot-qemu packages respectively. The usage looks like -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf I think in this case the first file is a minimal build of OpenSBI that likely just initializes enough hardware before handing control to an arbitrary payload - in this case, u-boot.

On Wed, Mar 22, 2023 at 10:36:20AM -0300, Daniel Henrique Barboza wrote: > I'm not sure if the OS overwrites the firmware when running bare metal. Usually > they provide different OS images for QEMU/libvirt and bare metal systems, probably > to account for these differences. Baking the firmware in the kernel like Fedora > Rawhide is doing was important a few years ago when RISC-V QEMU wasn't loading > the firmware by default, but now it's getting in the way. > > All this said, having a look at the recently updated Fedora for RISC-V wiki, the > instructions for booting with libvirt and QEMU are different. libvirt [1] is using > the '-bios none' attribute with 'virt-install' + a Fedora Rawhide image, but > QEMU instructions [2] doesn't use '-bios none' and it's using Fedora 37. The libvirt instructions in that page seem to have been updated in the Fedora 30 timeframe, whereas the QEMU instructions appear to be current as of Fedora 37. > At first I don't see any particular reason of why this Fedora 37 image would work > on QEMU and not on libvirt. So what t I'll do now is do some testings with libvirt > and Fedora 37. If it works then this series becomes way less important. The libvirt instructions tell you to use --boot kernel=Fedora-Developer-Rawhide-*-fw_payload-uboot-qemu-virt-smode.elf --qemu-commandline='-bios none' (aka -bios none -kernel foo) while the QEMU ones suggest -bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin -device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 Those are completely different approaches to booting the guest OS. The latter is much saner IMO, because it keeps the firmware under control of the host (as is the case for SeaBIOS/edk2) instead of mixing the kernel and the firmware and requiring guest-specific files to be extracted from the disk image before being able to boot. libvirt already provides the ability to pass arbitrary paths to -bios via the <loader type='rom'> element, so making the first part work should just be a matter of pointing it to the correct path. We don't seem to have the -device loader part wired up though, so that would need to be added. Probably as an additional attribute, in the vein of nvram.template? The address would have to be specified as well. Note that the firmware descriptor format already supports u-boot as the firmware type. So in the long run ideally you'd only need to specify <os firmware='uboot'> and, assuming the uboot-images-riscv64 package is installed on the host, everything should just work.

On Tue, May 23, 2023 at 11:59:41AM +0100, Richard W.M. Jones wrote: > I just came across this thread while trying to update the libvirt > instructions on that page. Specifically I need to add these to the > qemu command line: > > -bios /path/to/u-boot-spl.bin > -device loader,file=/path/to/u-boot.itb,addr=0x80200000 I recommend going through the entire thread if you haven't already, it's full of interesting details. And it would be useful to have another pair of eyes go over it, just in case something was missed :) Anyway, we found that the -device loader incantation above is effectively equivalent to -kernel, so you can just use that instead. Both -bios and -kernel are exposed by libvirt through XML elements of the same names. Further, since QEMU loads OpenSBI as -bios by default these days, you don't even need that part and can just use -kernel to point to the u-boot.bin file. In this case, you should use the file that would be installed as

/usr/share/uboot/qemu-riscv64_smode/u-boot.bin by the uboot-images-riscv64 Fedora package.

> Was any change made to libvirt / virt-install to make this possible? Not yet. I would love to make booting a riscv64 VM a simple case of adding <os firmware='u-boot'/> to the domain XML, and I'm more than willing to write the necessary libvirt patches, but a couple of things need to happen first: * the uboot-images-riscv64 package needs to be made available on non-riscv64 architectures; * a JSON firmware descriptor needs to be added to said package, pointing to /usr/share/uboot/qemu-riscv64_smode/u-boot.bin. The first part might be tricky, as it will probably involve a cross-compiler, but other distributions seem to have managed to do it already so I don't expect Fedora to face significant roadblocks. For the second part, something along the lines of # /usr/share/qemu/firmware/30-uboot-riscv64-smode.json { "interface-types": [ "uboot" ], "mapping": { "device": "kernel", "filename": /usr/share/uboot/qemu-riscv64_smode/u-boot.bin" }, "targets": [ { "architecture": "riscv64", "machines": [ "virt" ] } ] } should do the trick, but note that I haven't tested it :) My only question is why the u-boot-spl.bin/u-boot.itb approach was chosen to be documented in the Fedora wiki? Is there something that I've missed and that makes it superior, or can we follow along with what everyone else is doing? A question for David, perhaps.

> For the benefit of the search engine gods, this works for now: > > # virt-install --import --memory 8192 -n fedora-37-riscv \ > --arch riscv64 --vcpus 8 \ > --disk fedora-37-riscv.qcow2,format=qcow2 \ > --osinfo fedora37 \ > --qemu-commandline=' -bios /path/to/u-boot-spl.bin -device loader,file=/path/to/u-boot.itb,addr=0x80200000 '

Based on the information above, using --boot kernel=/path/to/u-boot.bin should work... > 'Course you have to disable SELinux ... ... without requiring this :) -- Andrea Bolognani / Red Hat / Virtualization

On Tue, May 23, 2023 at 04:05:04PM +0300, David Abdurachmanov wrote: > We run QEMU in a similar way as any other board. > > So it's U-Boot SPL, which then loads U-Boot ITB (which typically > contains U-Boot proper, OpenSBI FW_DYNAMIC generic binary, and DTB). > U-Boot SPL transfers control to OpenSBI and tells it how to load the > next stage (i.e U-Boot proper). Is there any documentation on how RISC-V boots now? And about the eventual goal that we're aiming for?

> > > For the benefit of the search engine gods, this works for now: > > > > > > # virt-install --import --memory 8192 -n fedora-37-riscv \ > > > --arch riscv64 --vcpus 8 \ > > > --disk fedora-37-riscv.qcow2,format=qcow2 \ > > > --osinfo fedora37 \ > > > --qemu-commandline=' -bios /path/to/u-boot-spl.bin -device loader,file=/path/to/u-boot.itb,addr=0x80200000 ' > > This doesn't disable Sv48 and Sv57. I don't know the overall status, > but at least Golang 1.20 has a fix to support anything above Sv39. > > Not sure about other runtimes that do pointer tagging. > > See: https://bugs.launchpad.net/ubuntu/+source/linux-riscv/+bug/1991790 Interesting and informative bug. I guess I'm not using any golang binaries, as the guest totally seems to work fine. I'm a bit confused how this kernel detail leaks into userspace though. I thought userspace can use all 64 bits in pointers.

> Simply put, older disk images on newer QEMU versions might not work properly. > > Note that we might want to switch to EDK2 in the future for QEMU, and > that probably will use two 32MiB pflash devices in virt machine. I > have seen, but haven't tested QEMU + EDK2 patches. Rich. > Cheers, > david > > > > > Based on the information above, using > > > > --boot kernel=/path/to/u-boot.bin > > > > should work... > > > > > 'Course you have to disable SELinux ... > > > > ... without requiring this :) > > > > -- > > Andrea Bolognani / Red Hat / Virtualization > > -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

On Tue, May 23, 2023 at 04:05:04PM +0300, David Abdurachmanov wrote: > On Tue, May 23, 2023 at 3:18 PM Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > On Tue, May 23, 2023 at 11:59:41AM +0100, Richard W.M. Jones wrote: > > > I just came across this thread while trying to update the libvirt > > > instructions on that page. Specifically I need to add these to the > > > qemu command line: > > > > > > -bios /path/to/u-boot-spl.bin > > > -device loader,file=/path/to/u-boot.itb,addr=0x80200000 > > > > Anyway, we found that the -device loader incantation above is > > effectively equivalent to -kernel, so you can just use that instead. > > Both -bios and -kernel are exposed by libvirt through XML elements of > > the same names. > > > > Further, since QEMU loads OpenSBI as -bios by default these days, you > > don't even need that part and can just use -kernel to point to the > > u-boot.bin file. In this case, you should use the file that would be > > installed as > > We run QEMU in a similar way as any other board. You mean physical boards?

> So it's U-Boot SPL, which then loads U-Boot ITB (which typically > contains U-Boot proper, OpenSBI FW_DYNAMIC generic binary, and DTB). > U-Boot SPL transfers control to OpenSBI and tells it how to load the > next stage (i.e U-Boot proper). That sounds a bit roundabout when you consider that QEMU automatically loads OpenSBI, and that in turn knows to jump to a S-Mode payload such as the Linux kernel or an S-Mode build of U-Boot. Put it another way: do you have any objections to loading qemu-riscv64_smode/u-boot.bin via -kernel as the default boot strategy for riscv64 VMs? That's what every OS other than Fedora already recommends doing.

Loading the SPL and ITB separately would still be possible, of course. I'm simply talking about the default experience.

> > /usr/share/uboot/qemu-riscv64_smode/u-boot.bin > > > > by the uboot-images-riscv64 Fedora package. > > This is noarch package, and thus available on all arches. This package > is only available in Fedora/RISCV Koji for now. You're right! I got confused because "riscv64" is in the name O:-) Is there ongoing work to move this package to Fedora proper? In terms of user experience, having to download disk images from a third-party is mostly fine, but installing third-party RPMs on the host... Not so much. Having this in Fedora proper would make it a fully self-contained host for RISC-V TCG VMs.

> > > For the benefit of the search engine gods, this works for now: > > > > > > # virt-install --import --memory 8192 -n fedora-37-riscv \ > > > --arch riscv64 --vcpus 8 \ > > > --disk fedora-37-riscv.qcow2,format=qcow2 \ > > > --osinfo fedora37 \ > > > --qemu-commandline=' -bios /path/to/u-boot-spl.bin -device loader,file=/path/to/u-boot.itb,addr=0x80200000 ' > > This doesn't disable Sv48 and Sv57. I don't know the overall status, > but at least Golang 1.20 has a fix to support anything above Sv39. > > Not sure about other runtimes that do pointer tagging. > > See: https://bugs.launchpad.net/ubuntu/+source/linux-riscv/+bug/1991790 IIUC that's something that needs to be handled in the kernel? Or do we need to do something at the QEMU/libvirt/virt-install level to make things work?

> Simply put, older disk images on newer QEMU versions might not work properly. I'm not too concerned about this. In the long run, it will simply not matter.

> Note that we might want to switch to EDK2 in the future for QEMU, and > that probably will use two 32MiB pflash devices in virt machine. I > have seen, but haven't tested QEMU + EDK2 patches. Yup, there's some discussion about this very topic happening right now on the QEMU mailing list. I'm actively involved in it and it looks like the necessary changes might be merged soon.

I'm not sure we can count on EFI-enabled RISC-V disk images popping up very quickly though, so I would like to improve the situation for the disk images that are out there right now and need U-Boot to run.

-- Andrea Bolognani / Red Hat / Virtualization

On Tue, May 23, 2023 at 06:32:07PM +0300, David Abdurachmanov wrote: > On Tue, May 23, 2023 at 6:12 PM Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > > > On Tue, May 23, 2023 at 04:05:04PM +0300, David Abdurachmanov wrote: > > > On Tue, May 23, 2023 at 3:18 PM Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > > > On Tue, May 23, 2023 at 11:59:41AM +0100, Richard W.M. Jones wrote: > > > > > I just came across this thread while trying to update the libvirt > > > > > instructions on that page. Specifically I need to add these to the > > > > > qemu command line: > > > > > > > > > > -bios /path/to/u-boot-spl.bin > > > > > -device loader,file=/path/to/u-boot.itb,addr=0x80200000 > > > > > > > > Anyway, we found that the -device loader incantation above is > > > > effectively equivalent to -kernel, so you can just use that instead. > > > > Both -bios and -kernel are exposed by libvirt through XML elements of > > > > the same names. > > > > > > > > Further, since QEMU loads OpenSBI as -bios by default these days, you > > > > don't even need that part and can just use -kernel to point to the > > > > u-boot.bin file. In this case, you should use the file that would be > > > > installed as > > > > > > We run QEMU in a similar way as any other board. > > > > You mean physical boards? > > Yes > > > > > > So it's U-Boot SPL, which then loads U-Boot ITB (which typically > > > contains U-Boot proper, OpenSBI FW_DYNAMIC generic binary, and DTB). > > > U-Boot SPL transfers control to OpenSBI and tells it how to load the > > > next stage (i.e U-Boot proper). > > > > That sounds a bit roundabout when you consider that QEMU > > automatically loads OpenSBI, and that in turn knows to jump to a > > S-Mode payload such as the Linux kernel or an S-Mode build of U-Boot. > > > > Put it another way: do you have any objections to loading > > qemu-riscv64_smode/u-boot.bin via -kernel as the default boot > > strategy for riscv64 VMs? That's what every OS other than Fedora > > already recommends doing. > > We typically use a non-release version of OpenSBI, because it has the > latest hardware support and maybe some fixes. > > For libvirt users (out-of-the box experience) that's the correct approach. > > > > > Loading the SPL and ITB separately would still be possible, of > > course. I'm simply talking about the default experience. > > Yeah, this default is sane. > > > > > > > /usr/share/uboot/qemu-riscv64_smode/u-boot.bin > > > > > > > > by the uboot-images-riscv64 Fedora package. > > > > > > This is noarch package, and thus available on all arches. This package > > > is only available in Fedora/RISCV Koji for now. > > > > You're right! I got confused because "riscv64" is in the name O:-) > > > > Is there ongoing work to move this package to Fedora proper? In terms > > of user experience, having to download disk images from a third-party > > is mostly fine, but installing third-party RPMs on the host... Not so > > much. Having this in Fedora proper would make it a fully > > self-contained host for RISC-V TCG VMs. > > It could go as-is, but most likely will need to change. So far we > relied on a single OpenSBI FW_DYNAMIC generic image, but that changes > starting with JH7110 which has a different (i.e. non default) load > address. Thus we might need to build OpenSBI per board (or similar), > and pick the right one while building U-Boot. > > Unknown (at least for me) how things will work on TH1520 based boards too. > > One thing I am sure about is that things in the SPEC file most likely > will change once more boards from multiple vendors are supported. > > We can push the current thing as-is if you want. The "hump" here is getting it through the Fedora approval process. Once in we can add additional features very easily. If you want me to do this let me know, I can start right away (still fiddling around with benchmarking that qemu instance so I'm not fully occupied right now).

Rich. > > > > > > > For the benefit of the search engine gods, this works for now: > > > > > > > > > > # virt-install --import --memory 8192 -n fedora-37-riscv \ > > > > > --arch riscv64 --vcpus 8 \ > > > > > --disk fedora-37-riscv.qcow2,format=qcow2 \ > > > > > --osinfo fedora37 \ > > > > > --qemu-commandline=' -bios /path/to/u-boot-spl.bin -device loader,file=/path/to/u-boot.itb,addr=0x80200000 ' > > > > > > This doesn't disable Sv48 and Sv57. I don't know the overall status, > > > but at least Golang 1.20 has a fix to support anything above Sv39. > > > > > > Not sure about other runtimes that do pointer tagging. > > > > > > See: https://bugs.launchpad.net/ubuntu/+source/linux-riscv/+bug/1991790 > > > > IIUC that's something that needs to be handled in the kernel? Or do > > we need to do something at the QEMU/libvirt/virt-install level to > > make things work? > > Basically two workarounds were created: > - Ability to control stap mode in QEMU (basically options to > enable/disable Sv39, Sv48, Sv57). > - Ability to disable that on the kernel side. > > I think the 1st one to arrive was QEMU. > > IIRC x86_64 and aarch64 are smart in the way they give virtual > addresses to user space. Which is not the case for riscv (yet). > > On the kernel side, that's part of 6.4 kernel (not released yet). > > > > > > Simply put, older disk images on newer QEMU versions might not work properly. > > > > I'm not too concerned about this. In the long run, it will simply not > > matter. > > True. > > > > > > Note that we might want to switch to EDK2 in the future for QEMU, and > > > that probably will use two 32MiB pflash devices in virt machine. I > > > have seen, but haven't tested QEMU + EDK2 patches. > > > > Yup, there's some discussion about this very topic happening right > > now on the QEMU mailing list. I'm actively involved in it and it > > looks like the necessary changes might be merged soon. > > Yup. I noticed your name on the emails. Good to know you are involved. > > > > > I'm not sure we can count on EFI-enabled RISC-V disk images popping > > up very quickly though, so I would like to improve the situation for > > the disk images that are out there right now and need U-Boot to run. > > EFI works and some are already using it with U-Boot IIRC. > Even smaller SBCs (e.g. based on StarFive JH7110) gained EDK2 port > (probably not upstreamed yet). > > https://github.com/starfive-tech/edk2/releases > > So even sub-100 USD boards will get EDK2 :) Of course, that doesn't incl. ACPI. > > Cheers, > david > > > > > -- > > Andrea Bolognani / Red Hat / Virtualization > > -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

On Tue, May 23, 2023 at 04:05:04PM +0300, David Abdurachmanov wrote: > On Tue, May 23, 2023 at 3:18 PM Andrea Bolognani <abologna(a)redhat.com&gt; wrote: > > On Tue, May 23, 2023 at 11:59:41AM +0100, Richard W.M. Jones wrote: > > > I just came across this thread while trying to update the libvirt > > > instructions on that page. Specifically I need to add these to the > > > qemu command line: > > > > > > -bios /path/to/u-boot-spl.bin > > > -device loader,file=/path/to/u-boot.itb,addr=0x80200000 > > > > Anyway, we found that the -device loader incantation above is > > effectively equivalent to -kernel, so you can just use that instead. > > Both -bios and -kernel are exposed by libvirt through XML elements of > > the same names. > > > > Further, since QEMU loads OpenSBI as -bios by default these days, you > > don't even need that part and can just use -kernel to point to the > > u-boot.bin file. In this case, you should use the file that would be > > installed as > > We run QEMU in a similar way as any other board. You mean physical boards? > So it's U-Boot SPL, which then loads U-Boot ITB (which typically > contains U-Boot proper, OpenSBI FW_DYNAMIC generic binary, and DTB). > U-Boot SPL transfers control to OpenSBI and tells it how to load the > next stage (i.e U-Boot proper). That sounds a bit roundabout when you consider that QEMU automatically loads OpenSBI, and that in turn knows to jump to a S-Mode payload such as the Linux kernel or an S-Mode build of U-Boot. Put it another way: do you have any objections to loading qemu-riscv64_smode/u-boot.bin via -kernel as the default boot strategy for riscv64 VMs? That's what every OS other than Fedora already recommends doing. Loading the SPL and ITB separately would still be possible, of course. I'm simply talking about the default experience. > > /usr/share/uboot/qemu-riscv64_smode/u-boot.bin > > > > by the uboot-images-riscv64 Fedora package. > > This is noarch package, and thus available on all arches. This package > is only available in Fedora/RISCV Koji for now. You're right! I got confused because "riscv64" is in the name O:-) Is there ongoing work to move this package to Fedora proper? In terms of user experience, having to download disk images from a third-party is mostly fine, but installing third-party RPMs on the host... Not so much. Having this in Fedora proper would make it a fully self-contained host for RISC-V TCG VMs.

> > > For the benefit of the search engine gods, this works for now: > > > > > > # virt-install --import --memory 8192 -n fedora-37-riscv \ > > > --arch riscv64 --vcpus 8 \ > > > --disk fedora-37-riscv.qcow2,format=qcow2 \ > > > --osinfo fedora37 \ > > > --qemu-commandline=' -bios /path/to/u-boot-spl.bin -device loader,file=/path/to/u-boot.itb,addr=0x80200000 ' > > This doesn't disable Sv48 and Sv57. I don't know the overall status, > but at least Golang 1.20 has a fix to support anything above Sv39. > > Not sure about other runtimes that do pointer tagging. > > See: https://bugs.launchpad.net/ubuntu/+source/linux-riscv/+bug/1991790 IIUC that's something that needs to be handled in the kernel? Or do we need to do something at the QEMU/libvirt/virt-install level to make things work?

> Simply put, older disk images on newer QEMU versions might not work properly. I'm not too concerned about this. In the long run, it will simply not matter. > Note that we might want to switch to EDK2 in the future for QEMU, and > that probably will use two 32MiB pflash devices in virt machine. I > have seen, but haven't tested QEMU + EDK2 patches. Yup, there's some discussion about this very topic happening right now on the QEMU mailing list. I'm actively involved in it and it looks like the necessary changes might be merged soon. I'm not sure we can count on EFI-enabled RISC-V disk images popping up very quickly though, so I would like to improve the situation for the disk images that are out there right now and need U-Boot to run.