Re: [libvirt PATCH V2 0/4] add loongarch support for libvirt

Hi Andrea:     I'm sorry for taking so long to reply you! Hi Philippe:     When developing libvirt on loongarch, we encountered some problems related to pflash. libvirt and qemu met some difficulties in the coordination of UEFI loading. I think we need your suggestions and opinions on the solution. > On Fri, Jan 12, 2024 at 02:32:43PM +0800, Xianglai Li wrote: >> Hello, Everyone: >> This patch series adds libvirt support for loongarch.Although the bios >> path and name has not been officially integrated into qemu and we think >> there are still many shortcomings, we try to push a version of patch to >> the community according to the opinions of the community, hoping to >> listen to everyone's opinions. Anyway we have a version of libvirt that >> supports loongarch. >> >> You can also get libvirt's patch from the link below: >> https://gitlab.com/lixianglai/libvirt >> branch: loongarch >> >> Since the patch associated with loongarch has not yet been submitted to >> the virt-manager community, we are providing a temporary patch with >> loongarch for the time being patch's virt-manager, the open source work >> of virt-manager adding loongarch will be followed up later or >> synchronized with the open source libvirt. >> >> You can get the virt-manager code with loongarch patch from the link below: >> https://github.com/lixianglai/virt-manager >> branch: loongarch > It's really great that you're considering the entire stack, all the > way up to virt-manager, while working on this! :) Oh! Thanks! :) >> loongarch's virtual machine bios is not yet available in qemu, so you can get it from the following link >> https://github.com/lixianglai/LoongarchVirtFirmware >> (Note: You should clone the repository using git instead of downloading the file via wget or you'll get xml) >> We named the bios QEMU_EFI.fd, QEMU_VARS.fd is used to store pflash images of non-volatile >> variables.After installing qemu-system-loongarch64,You can install the loongarch bios by executing the script >> install-loongarch-virt-firmware.sh > To clarify, loongarch support has been merged into upstream edk2 > already, right? And the contents of this repository are just for > convenience, since loongarch builds of edk2 have generally not made > it into distros yet? I think I might have asked about this already, > sorry O:-) Don't mention it. All right, The loongarch support has been merged into upstream edk2 already. And you can see a detailed description of loongarch uefi in the link below: https://github.com/tianocore/edk2-platforms/blob/master/Platform/Loongson... this repository are just for convenience. :)

> Anyway, I fetched and installed this. The firmware descriptor looks > like: > > { > "interface-types": [ > "uefi" > ], > "mapping": { > "device": "memory", > "filename": "/usr/share/edk2/loongarch64/QEMU_EFI.fd" > }, > "targets": [ > { > "architecture": "loongarch64", > "machines": [ > "virt", > "virt-*" > ] > } > ], > "features": [ > "acpi" > ] > } > > This is not what I expected: specifically, it results in libvirt > generating > > -bios /usr/share/edk2/loongarch64/QEMU_EFI.fd > > So only one of the two files is used, in read-only mode, and there is > no persistent NVRAM storage that the guest can use. > > This is what I expected instead: > > { > "interface-types": [ > "uefi" > ], > "mapping": { > "device": "flash", > "mode": "split", > "executable": { > "filename": "/usr/share/edk2/loongarch64/QEMU_EFI.fd", > "format": "raw" > }, > "nvram-template": { > "filename": "/usr/share/edk2/loongarch64/QEMU_VARS.fd", > "format": "raw" > } > }, > "targets": [ > { > "architecture": "loongarch64", > "machines": [ > "virt", > "virt-*" > ] > } > ], > "features": [ > "acpi" > ] > } > > I've tried installing such a descriptor and libvirt picks it up, > resulting in the following guest configuration: > > <os firmware='efi'> > <type arch='loongarch64' machine='virt'>hvm</type> > <firmware> > <feature enabled='no' name='enrolled-keys'/> > <feature enabled='no' name='secure-boot'/> > </firmware> > <loader readonly='yes' > type='pflash'>/usr/share/edk2/loongarch64/QEMU_EFI.fd</loader> > <nvram template='/usr/share/edk2/loongarch64/QEMU_VARS.fd'>/var/lib/libvirt/qemu/nvram/guest_VARS.fd</nvram> > <boot dev='hd'/> > </os> > > which in turn produces the following QEMU command line options: > > -blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_EFI.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' > -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' > -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/guest_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' > -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' > > Unfortunately, with this configuration the guest fails to start: > > qemu-system-loongarch64: Property 'virt-machine.pflash0' not found > > This error message looked familiar to me, as it is the same that I > hit when trying out UEFI support on RISC-V roughly a year ago[1]. In > this case, however, it seems that the issue runs deeper: it's not > just that the flash devices are not wired up to work as blockdevs, > but even the old -drive syntax doesn't work. > > Looking at the QEMU code, it appears that the loongarch/virt machine > only creates a single pflash device and exposes it via -bios. So it > seems that there is simply no way to achieve the configuration that > we want. > > I think that this is something that needs to be addressed as soon as > possible. In the long run, guest-accessible NVRAM storage is a must, > and I'm not sure it would make a lot of sense to merge loongarch > support into libvirt until the firmware situation has been sorted out > in the lower layers. In the qemu code, loongarch virt machine does only create a pflash, which is used for nvram, and uefi code is loaded by rom. In summary, loongarch virt machine can use nvram with the following command: ------------------------------------------------------------------------------------------------------- /qemu-system-loongarch64 \// //-m 8G \// //-smp 4 \// //-cpu la464   \// //-blockdev '{"driver":"file","filename":"./QEMU_VARS-pflash.raw","node-name":"libvirt-pflash0-storage","auto-read-only":false,"discard":"unmap"}' \// //-blockdev '{"node-name":"libvirt-pflash0-format","read-only":false,"driver":"raw","file":"libvirt-pflash0-storage"}' \// //-machine virt,pflash=libvirt-pflash0-format  \// //-snapshot \// //-bios ./QEMU_EFI.fd  \// //-serial stdio/ ------------------------------------------------------------------------------------------------------- This is really a big difference from the following boot method, and it still looks weird. ------------------------------------------------------------------------------------------------------- /-blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_EFI.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/guest_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}'/ ------------------------------------------------------------------------------------------------------- However, during the development of qemu loongarch, we also used a RISCV-like solution to create two pflash, but the qemu community suggested that we put uefi code in rom for the following reasons: https://lore.kernel.org/qemu-devel/2f381d06-842f-ac8b-085c-0419675a4872@l... " Since you are starting a virtual machine from scratch, you should take the opportunity to learn from other early mistakes. X86 ended that way due to 1/ old firmwares back-compability and 2/ QEMU pflash block protections not being implemented. IIUC if we were starting with a UEFI firmware today, the layout design (still using QEMU) would be to map the CODE area in a dumb ROM device, and the VARSTORE area in a PFlash device. Since Virt machines don't need to use Capsule update, having the CODE area in ROM drastically simplifies the design and maintainance. " Well, anyway, now that we have an issue with qemu loongarch using nvram that is incompatible with libvirt, here I have come up with two solutions to solve this problem: Option 1: If the interface type "rom-uefi" is added and the device type "rom-flash" is added, the json file should be written like this: ------------------------------------------------------------------------------------------------------- /{// //   "interface-types": [// ////"rom-uefi"// //   ],// //   "mapping": {// ////"device": "rom-flash",// //     "executable": {// //       "filename": "/usr/share/edk2/loongarch64/QEMU_EFI.fd",// //       "format": "raw"// //     },// //     "nvram-template": {// //       "filename": "/usr/share/edk2/loongarch64/QEMU_VARS.fd",// //       "format": "raw"// //     }// //   },// //   "targets": [// //     {// //       "architecture": "loongarch64",// //       "machines": [// //         "virt",// //         "virt-*"// //       ]// //     }// //   ],// //   "features": [// //       "acpi"// //   ]/  ------------------------------------------------------------------------------------------------------- Then add the parsing of the new interface types in libvirt and load QEMU_CODE.fd as -bios and QEMU_VARS.fd as nvram when creating the command line, generating commands like the following:  ------------------------------------------------------------------------------------------------------- /qemu-system-loongarch64 \// //-m 8G \// //-smp 1 \// //-cpu la464   \// //-blockdev '{"driver":"file","filename":"////usr/share/edk2/loongarch64/QEMU_VARS.fd/","node-name":"libvirt-pflash0-storage","auto-read-only":false,"discard":"unmap"}' \// //-blockdev '{"node-name":"libvirt-pflash0-format","read-only":false,"driver":"raw","file":"libvirt-pflash0-storage"}' \// //-machine virt,pflash=libvirt-pflash0-format  \// //-snapshot \// //-bios /////usr/share/edk2/loongarch64///QEMU_EFI.fd \// //-serial stdio/ ------------------------------------------------------------------------------------------------------- Option 2: Solution 2 mainly starts from qemu. Now the rom that bios is loaded into is a memory region that cannot be configured with attributes, so we imagine abstracting rom as a device, creating it during machine initialization and setting "pflash0" attribute for it. Then create a pflash and set its property to "pflash1", so our startup command will look like this:  ------------------------------------------------------------------------------------------------------- /qemu-system-loongarch64 \// //-m 8G \// //-smp 1 \// //-cpu la464   \// //-blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_EFI.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \// //-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \// //-blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \// //-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \// //-machine virt,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \// //-snapshot \// //-serial stdio/  ------------------------------------------------------------------------------------------------------- This way, without modifying libvirt, QEMU_CODE.fd can be loaded into the rom, but it is still a little strange that it is clearly rom but set a "pflash0" attribute, which can be confusing. >> Since there is no fedora operating system that supports the loongarch >> architecture, you can find an iso that supports loongarch at the link >> below for testing purposes: >> https://github.com/fedora-remix-loongarch/releases-info >> >> Well, if you have completed the above steps I think you can now install loongarch virtual machine, >> you can install it through the virt-manager graphical interface, or install it through vrit-install, >> here is an example of installing it using virt-install: >> >> virt-install \ >> --virt-type=qemu \ >> --name loongarch-test \ >> --memory 4096 \ >> --vcpus=4 \ >> --arch=loongarch64 \ >> --boot cdrom \ >> --disk device=cdrom,bus=scsi,path=/root/livecd-fedora-mate-4.loongarch64.iso \ >> --disk path=/var/lib/libvirt/images/debian12-loongarch64.qcow2,size=10,format=qcow2,bus=scsi \ >> --network network=default \ >> --osinfo archlinux \ >> --video=virtio \ >> --graphics=vnc,listen=0.0.0.0 > Regardless of what I said above, this actually worked quite well! I > was able to boot the MATE image and, ignoring the expected slowness > caused by emulation, things were generally fairly smooth. > > Note that I tried the LXDE image first, expecting it to be a bit > smaller and faster, but in that case I wasn't able to get a > successful boot: edk2 loaded up just fine, but the guest OS was never > started and I was dropped into the edk2 shell instead. Maybe that's a > known issue, but I thought I'd point it out. Thank you very much for pointing out the problem, lxde iso is missing efiboot.img file so that the virtual machine UEFI can not recognize efi partition caused by the problem has been notified to the relevant personnel, I believe that the next version of libvirt patch will be shipped to use lxde iso. Thanks, Xianglai. > [1]https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg06258.html