Hi All,

Added my notes below.

Thanks,
Chao

On 2024/1/30 16:59, lixianglai wrote:

Hi Andrea:

I'm sorry for taking so long to reply you!

Hi Philippe:

When developing libvirt on loongarch, we encountered some problems related to pflash.

libvirt and qemu met some difficulties in the coordination of UEFI loading.

I think we need your suggestions and opinions on the solution.

On Fri, Jan 12, 2024 at 02:32:43PM +0800, Xianglai Li wrote:

Hello, Everyone:
  This patch series adds libvirt support for loongarch.Although the bios
path and name has not been officially integrated into qemu and we think
there are still many shortcomings, we try to push a version of patch to
the community according to the opinions of the community, hoping to
listen to everyone's opinions. Anyway we have a version of libvirt that
supports loongarch.

  You can also get libvirt's patch from the link below:
https://gitlab.com/lixianglai/libvirt
branch: loongarch

  Since the patch associated with loongarch has not yet been submitted to
the virt-manager community, we are providing a temporary patch with
loongarch for the time being patch's virt-manager, the open source work
of virt-manager adding loongarch will be followed up later or
synchronized with the open source libvirt.

  You can get the virt-manager code with loongarch patch from the link below:
https://github.com/lixianglai/virt-manager
branch: loongarch

It's really great that you're considering the entire stack, all the
way up to virt-manager, while working on this! :)

Oh! Thanks!

:)

  loongarch's virtual machine bios is not yet available in qemu, so you can get it from the following link
https://github.com/lixianglai/LoongarchVirtFirmware
(Note: You should clone the repository using git instead of downloading the file via wget or you'll get xml)
We named the bios QEMU_EFI.fd, QEMU_VARS.fd is used to store pflash images of non-volatile
variables.After installing qemu-system-loongarch64,You can install the loongarch bios by executing the script
install-loongarch-virt-firmware.sh

To clarify, loongarch support has been merged into upstream edk2
already, right? And the contents of this repository are just for
convenience, since loongarch builds of edk2 have generally not made
it into distros yet? I think I might have asked about this already,
sorry O:-)

Don't mention it.

All right, The loongarch support has been merged into upstream edk2 already.

And you can see a detailed description of loongarch uefi in the link below:

https://github.com/tianocore/edk2-platforms/blob/master/Platform/Loongson/LoongArchQemuPkg/Readme.md

this repository are just for convenience. :)

BTW, LoongArchVirt under OvmfPkg is being committed to upstream, I guess it will be merged before the stable202402, please refer to:

https://bugzilla.tianocore.org/show_bug.cgi?id=4584

https://edk2.groups.io/g/devel/message/114526

https://github.com/tianocore/edk2/pull/5208

Most maintainer have completed the review.

Anyway, I fetched and installed this. The firmware descriptor looks
like:

  {
     "interface-types": [
       "uefi"
     ],
     "mapping": {
       "device": "memory",
       "filename": "/usr/share/edk2/loongarch64/QEMU_EFI.fd"
     },
     "targets": [
       {
         "architecture": "loongarch64",
         "machines": [
           "virt",
           "virt-*"
         ]
       }
     ],
     "features": [
         "acpi"
     ]
   }

This is not what I expected: specifically, it results in libvirt
generating

  -bios /usr/share/edk2/loongarch64/QEMU_EFI.fd

So only one of the two files is used, in read-only mode, and there is
no persistent NVRAM storage that the guest can use.

This is what I expected instead:

  {
     "interface-types": [
       "uefi"
     ],
     "mapping": {
       "device": "flash",
       "mode": "split",
       "executable": {
         "filename": "/usr/share/edk2/loongarch64/QEMU_EFI.fd",
         "format": "raw"
       },
       "nvram-template": {
         "filename": "/usr/share/edk2/loongarch64/QEMU_VARS.fd",
         "format": "raw"
       }
     },
     "targets": [
       {
         "architecture": "loongarch64",
         "machines": [
           "virt",
           "virt-*"
         ]
       }
     ],
     "features": [
         "acpi"
    ]
  }

I've tried installing such a descriptor and libvirt picks it up,
resulting in the following guest configuration:

  <os firmware='efi'>
    <type arch='loongarch64' machine='virt'>hvm</type>
    <firmware>
      <feature enabled='no' name='enrolled-keys'/>
      <feature enabled='no' name='secure-boot'/>
    </firmware>
    <loader readonly='yes'
type='pflash'>/usr/share/edk2/loongarch64/QEMU_EFI.fd</loader>
    <nvram template='/usr/share/edk2/loongarch64/QEMU_VARS.fd'>/var/lib/libvirt/qemu/nvram/guest_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

which in turn produces the following QEMU command line options:

  -blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_EFI.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}'
  -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}'
  -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/guest_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}'
  -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}'

Unfortunately, with this configuration the guest fails to start:

  qemu-system-loongarch64: Property 'virt-machine.pflash0' not found

This error message looked familiar to me, as it is the same that I
hit when trying out UEFI support on RISC-V roughly a year ago[1]. In
this case, however, it seems that the issue runs deeper: it's not
just that the flash devices are not wired up to work as blockdevs,
but even the old -drive syntax doesn't work.

Looking at the QEMU code, it appears that the loongarch/virt machine
only creates a single pflash device and exposes it via -bios. So it
seems that there is simply no way to achieve the configuration that
we want.

I think that this is something that needs to be addressed as soon as
possible. In the long run, guest-accessible NVRAM storage is a must,
and I'm not sure it would make a lot of sense to merge loongarch
support into libvirt until the firmware situation has been sorted out
in the lower layers.
In the qemu code, loongarch virt machine does only create a pflash,

which is used for nvram, and uefi code is loaded by rom.

In summary, loongarch virt machine can use nvram with the following command:

-------------------------------------------------------------------------------------------------------

qemu-system-loongarch64 \
-m 8G \
-smp 4 \
-cpu la464 \
-blockdev '{"driver":"file","filename":"./QEMU_VARS-pflash.raw","node-name":"libvirt-pflash0-storage","auto-read-only":false,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":false,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-machine virt,pflash=libvirt-pflash0-format \
-snapshot \
-bios ./QEMU_EFI.fd \
-serial stdio

-------------------------------------------------------------------------------------------------------

This is really a big difference from the following boot method, and it still looks weird.

-------------------------------------------------------------------------------------------------------
  -blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_EFI.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}'
  -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}'
  -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/guest_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}'
  -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}'
-------------------------------------------------------------------------------------------------------

However, during the development of qemu loongarch,

we also used a RISCV-like solution to create two pflash,

but the qemu community suggested that we put uefi code in rom for the following reasons:

https://lore.kernel.org/qemu-devel/2f381d06-842f-ac8b-085c-0419675a4872@linaro.org/

"
Since you are starting a virtual machine from scratch, you should take
the opportunity to learn from other early mistakes. X86 ended that way
due to 1/ old firmwares back-compability and 2/ QEMU pflash block
protections not being implemented. IIUC if we were starting with a
UEFI firmware today, the layout design (still using QEMU) would be
to map the CODE area in a dumb ROM device, and the VARSTORE area
in a PFlash device. Since Virt machines don't need to use Capsule
update, having the CODE area in ROM drastically simplifies the design
and maintainance.
"

Well, anyway, now that we have an issue with qemu loongarch using nvram that is incompatible with libvirt,

here I have come up with two solutions to solve this problem:

Option 1:

If the interface type "rom-uefi" is added and the device type "rom-flash" is added, the json file should be written like this:

-------------------------------------------------------------------------------------------------------

{
   "interface-types": [
     "rom-uefi"
   ],
   "mapping": {
     "device": "rom-flash",
     "executable": {
       "filename": "/usr/share/edk2/loongarch64/QEMU_EFI.fd",
       "format": "raw"
     },
     "nvram-template": {
       "filename": "/usr/share/edk2/loongarch64/QEMU_VARS.fd",
       "format": "raw"
     }
   },
   "targets": [
     {
       "architecture": "loongarch64",
       "machines": [
         "virt",
         "virt-*"
       ]
     }
   ],
   "features": [
       "acpi"
   ]

-------------------------------------------------------------------------------------------------------

Then add the parsing of the new interface types in libvirt and load QEMU_CODE.fd as -bios and QEMU_VARS.fd as nvram

when creating the command line, generating commands like the following:

-------------------------------------------------------------------------------------------------------

qemu-system-loongarch64 \
-m 8G \
-smp 1 \
-cpu la464   \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_VARS.fd","node-name":"libvirt-pflash0-storage","auto-read-only":false,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":false,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-machine virt,pflash=libvirt-pflash0-format \
-snapshot \
-bios /usr/share/edk2/loongarch64/QEMU_EFI.fd \
-serial stdio

-------------------------------------------------------------------------------------------------------

Option 2:

Solution 2 mainly starts from qemu. Now the rom that bios is loaded into is a memory region that cannot be configured with attributes,

so we imagine abstracting rom as a device, creating it during machine initialization and setting "pflash0" attribute for it.

Then create a pflash and set its property to "pflash1", so our startup command will look like this:

-------------------------------------------------------------------------------------------------------

qemu-system-loongarch64 \
-m 8G \
-smp 1 \
-cpu la464   \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_EFI.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/loongarch64/QEMU_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine virt,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-snapshot \
-serial stdio

-------------------------------------------------------------------------------------------------------

This way, without modifying libvirt, QEMU_CODE.fd can be loaded into the rom,

but it is still a little strange that it is clearly rom but set a "pflash0" attribute, which can be confusing.
  Since there is no fedora operating system that supports the loongarch
architecture, you can find an iso that supports loongarch at the link
below for testing purposes:
  https://github.com/fedora-remix-loongarch/releases-info

  Well, if you have completed the above steps I think you can now install loongarch virtual machine,
you can install it through the virt-manager graphical interface, or install it through vrit-install,
here is an example of installing it using virt-install:

virt-install  \
--virt-type=qemu \
--name  loongarch-test \
--memory 4096 \
--vcpus=4 \
--arch=loongarch64 \
--boot cdrom \
--disk device=cdrom,bus=scsi,path=/root/livecd-fedora-mate-4.loongarch64.iso \
--disk path=/var/lib/libvirt/images/debian12-loongarch64.qcow2,size=10,format=qcow2,bus=scsi \
--network network=default \
--osinfo archlinux   \
--video=virtio  \
--graphics=vnc,listen=0.0.0.0
Regardless of what I said above, this actually worked quite well! I
was able to boot the MATE image and, ignoring the expected slowness
caused by emulation, things were generally fairly smooth.

Note that I tried the LXDE image first, expecting it to be a bit
smaller and faster, but in that case I wasn't able to get a
successful boot: edk2 loaded up just fine, but the guest OS was never
started and I was dropped into the edk2 shell instead. Maybe that's a
known issue, but I thought I'd point it out.
Thank you very much for pointing out the problem,

lxde iso is missing efiboot.img file so that the virtual machine UEFI can not recognize efi partition

caused by the problem has been notified to the relevant personnel,

I believe that the next version of libvirt patch will be shipped to use lxde iso.

Thanks,

Xianglai.
[1] https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg06258.html