[PATCH 0/1] fix risc-v QEMU domains in non-RISC-V hosts
Hi, For some reason didn't notice this was broken until recently. We're missing an API in the RISC-V CPU driver to allow QEMU TCG risc-v domains to run in non-native hosts. Daniel Henrique Barboza (1): cpu_riscv64.c: add update() implementation src/cpu/cpu_riscv64.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) -- 2.40.0
At this moment it is not possible to launch a 'riscv64' domain of type 'qemu' (i.e. TCG) and machine 'virt' in a x86 host: $ sudo ./run tools/virsh start riscv-virt1 error: Failed to start domain 'riscv-virt1' error: this function is not supported by the connection driver: cannot update guest CPU for riscv64 architecture A guest CPU definition will be added if absent by qemuDomainSetDefaultCPU(): <cpu mode='custom' match='exact' check='none'> <model fallback='forbid'>rv64</model> </cpu> This def will then be updated in qemuProcessUpdateGuestCPU(), which calls virCPUUpdate(). The error above is being thrown by this function because the current RISC-V driver does not have an 'update' API implemented. There is no particular reason to not support TCG RISC-V guests in x86 or any other host arch, so let's add an 'update' API to the RISC-V driver. The code was copied from virCPUarmUpdate() from the ARM driver since it's a good enough implementation to get us started. Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com> --- src/cpu/cpu_riscv64.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/src/cpu/cpu_riscv64.c b/src/cpu/cpu_riscv64.c index c44bdeb291..9796f8c967 100644 --- a/src/cpu/cpu_riscv64.c +++ b/src/cpu/cpu_riscv64.c @@ -46,6 +46,32 @@ virCPURiscv64ValidateFeatures(virCPUDef *cpu G_GNUC_UNUSED) } +static int +virCPUriscvUpdate(virCPUDef *guest, + const virCPUDef *host, + bool relative) +{ + g_autoptr(virCPUDef) updated = virCPUDefCopyWithoutModel(guest); + + if (!relative || guest->mode != VIR_CPU_MODE_HOST_MODEL) + return 0; + + if (!host) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("unknown host CPU model")); + return -1; + } + + updated->mode = VIR_CPU_MODE_CUSTOM; + virCPUDefCopyModel(updated, host, true); + + virCPUDefStealModel(guest, updated, false); + guest->mode = VIR_CPU_MODE_CUSTOM; + guest->match = VIR_CPU_MATCH_EXACT; + + return 0; +} + struct cpuArchDriver cpuDriverRiscv64 = { .name = "riscv64", .arch = archs, @@ -54,6 +80,6 @@ struct cpuArchDriver cpuDriverRiscv64 = { .decode = NULL, .encode = NULL, .baseline = NULL, - .update = NULL, + .update = virCPUriscvUpdate, .validateFeatures = virCPURiscv64ValidateFeatures, }; -- 2.40.0
On Thu, Apr 27, 2023 at 06:04:10PM -0300, Daniel Henrique Barboza wrote:
At this moment it is not possible to launch a 'riscv64' domain of type 'qemu' (i.e. TCG) and machine 'virt' in a x86 host:
$ sudo ./run tools/virsh start riscv-virt1 error: Failed to start domain 'riscv-virt1' error: this function is not supported by the connection driver: cannot update guest CPU for riscv64 architecture
I get a different error message, specifically unsupported configuration: CPU mode 'custom' for riscv64 qemu domain on x86_64 host is not supported by hypervisor
A guest CPU definition will be added if absent by qemuDomainSetDefaultCPU():
No such function exists, did you mean qemuDomainDefSetDefaultCPU()?
<cpu mode='custom' match='exact' check='none'> <model fallback='forbid'>rv64</model> </cpu>
Note that I only started seeing the error message mentioned above after I manually added this snippet to my domain XML. If I don't do that, the <cpu> element doesn't show up at all and the VM can be started just fine.
struct cpuArchDriver cpuDriverRiscv64 = { .name = "riscv64", .arch = archs, @@ -54,6 +80,6 @@ struct cpuArchDriver cpuDriverRiscv64 = { .decode = NULL, .encode = NULL, .baseline = NULL, - .update = NULL, + .update = virCPUriscvUpdate, .validateFeatures = virCPURiscv64ValidateFeatures,
You should follow the naming convention already established for the file. I have not actually looked at the code yet. In theory the changes sound fine, but the current situation as you describe it differs enough from what I'm seeing here that I want to make sure we're on the same page before spending any time on that. Are you working off master? Do you have additional patches applied locally? The full XML for the VM that I'm using for testing is below. ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- <domain type='qemu'> <name>ubuntu-riscv</name> <uuid>586faf27-2bf8-49cb-9601-9a62fe3de28d</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>8</vcpu> <os> <type arch='riscv64' machine='virt'>hvm</type> <kernel>/var/lib/libvirt/images/riscv.u-boot.bin</kernel> <boot dev='hd'/> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-riscv64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/ubuntu-riscv.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='qemu-xhci'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:30:b4:bd'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='system-serial' port='0'> <model name='16550a'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <audio id='1' type='none'/> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </rng> </devices> </domain> -- Andrea Bolognani / Red Hat / Virtualization
On 4/28/23 12:40, Andrea Bolognani wrote:
On Thu, Apr 27, 2023 at 06:04:10PM -0300, Daniel Henrique Barboza wrote:
At this moment it is not possible to launch a 'riscv64' domain of type 'qemu' (i.e. TCG) and machine 'virt' in a x86 host:
$ sudo ./run tools/virsh start riscv-virt1 error: Failed to start domain 'riscv-virt1' error: this function is not supported by the connection driver: cannot update guest CPU for riscv64 architecture
I get a different error message, specifically
unsupported configuration: CPU mode 'custom' for riscv64 qemu domain on x86_64 host is not supported by hypervisor
Ha, I believe the patch is still relevant but I did a serious mess in the commit message :( The difference between what we're seeing is that I used a QEMU from riscv-to-apply.next from the maintainer tree [1] instead of master. That tree has the following patch: commit c13dfff2ded9cf2098a07e2426ffcad674602030 Author: Daniel Henrique Barboza <dbarboza@ventanamicro.com> Date: Tue Apr 11 15:35:10 2023 -0300 target/riscv: add query-cpy-definitions support With this QEMU, two things you're not seeing will start to happen: - virQEMUCapsGetMachineDefaultCPU() will return the 'rv64' CPU, and qemuDomainDefSetDefaultCPU() will add the <cpu> element during PostParse() time; - the error you're seeing from virQEMUCapsIsCPUModeSupported() won't happen because this: cpus = virQEMUCapsGetAccel(qemuCaps, type)->cpuModels; return cpus && cpus->ncpus > 0; won't return NULL anymore. You'll error out in the same point I am right now, in virCPUUpdate(), because there's no driver->update() for the RISC-V driver. I apologize for the confusion. I'll re-send the patch with a proper description: we need the update() implementation to support the <cpu> definition. I'll also make sure to test with QEMU master this time. Thanks, Daniel [1] https://github.com/alistair23/qemu/tree/riscv-to-apply.next
A guest CPU definition will be added if absent by qemuDomainSetDefaultCPU():
No such function exists, did you mean qemuDomainDefSetDefaultCPU()?
<cpu mode='custom' match='exact' check='none'> <model fallback='forbid'>rv64</model> </cpu>
Note that I only started seeing the error message mentioned above after I manually added this snippet to my domain XML.
If I don't do that, the <cpu> element doesn't show up at all and the VM can be started just fine.
struct cpuArchDriver cpuDriverRiscv64 = { .name = "riscv64", .arch = archs, @@ -54,6 +80,6 @@ struct cpuArchDriver cpuDriverRiscv64 = { .decode = NULL, .encode = NULL, .baseline = NULL, - .update = NULL, + .update = virCPUriscvUpdate, .validateFeatures = virCPURiscv64ValidateFeatures,
You should follow the naming convention already established for the file.
I have not actually looked at the code yet. In theory the changes sound fine, but the current situation as you describe it differs enough from what I'm seeing here that I want to make sure we're on the same page before spending any time on that. Are you working off master? Do you have additional patches applied locally?
The full XML for the VM that I'm using for testing is below.
----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< -----
<domain type='qemu'> <name>ubuntu-riscv</name> <uuid>586faf27-2bf8-49cb-9601-9a62fe3de28d</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>8</vcpu> <os> <type arch='riscv64' machine='virt'>hvm</type> <kernel>/var/lib/libvirt/images/riscv.u-boot.bin</kernel> <boot dev='hd'/> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-riscv64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/ubuntu-riscv.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='qemu-xhci'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:30:b4:bd'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='system-serial' port='0'> <model name='16550a'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <audio id='1' type='none'/> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </rng> </devices> </domain>
On Fri, Apr 28, 2023 at 02:15:04PM -0300, Daniel Henrique Barboza wrote:
On 4/28/23 12:40, Andrea Bolognani wrote:
On Thu, Apr 27, 2023 at 06:04:10PM -0300, Daniel Henrique Barboza wrote:
At this moment it is not possible to launch a 'riscv64' domain of type 'qemu' (i.e. TCG) and machine 'virt' in a x86 host:
$ sudo ./run tools/virsh start riscv-virt1 error: Failed to start domain 'riscv-virt1' error: this function is not supported by the connection driver: cannot update guest CPU for riscv64 architecture
I get a different error message, specifically
unsupported configuration: CPU mode 'custom' for riscv64 qemu domain on x86_64 host is not supported by hypervisor
Ha, I believe the patch is still relevant but I did a serious mess in the commit message :(
The difference between what we're seeing is that I used a QEMU from riscv-to-apply.next from the maintainer tree [1] instead of master.
I was thinking about libvirt's master branch rather than QEMU's, but that makes sense. I'll check out v2 after building QEMU from the appropriate branch.
commit c13dfff2ded9cf2098a07e2426ffcad674602030 Author: Daniel Henrique Barboza <dbarboza@ventanamicro.com> Date: Tue Apr 11 15:35:10 2023 -0300
target/riscv: add query-cpy-definitions support
Since this hasn't made its way into master yet, maybe you can still get the maintainer to s/cpy/cpu/ for you? :) -- Andrea Bolognani / Red Hat / Virtualization
On 5/3/23 06:07, Andrea Bolognani wrote:
On Fri, Apr 28, 2023 at 02:15:04PM -0300, Daniel Henrique Barboza wrote:
On 4/28/23 12:40, Andrea Bolognani wrote:
On Thu, Apr 27, 2023 at 06:04:10PM -0300, Daniel Henrique Barboza wrote:
At this moment it is not possible to launch a 'riscv64' domain of type 'qemu' (i.e. TCG) and machine 'virt' in a x86 host:
$ sudo ./run tools/virsh start riscv-virt1 error: Failed to start domain 'riscv-virt1' error: this function is not supported by the connection driver: cannot update guest CPU for riscv64 architecture
I get a different error message, specifically
unsupported configuration: CPU mode 'custom' for riscv64 qemu domain on x86_64 host is not supported by hypervisor
Ha, I believe the patch is still relevant but I did a serious mess in the commit message :(
The difference between what we're seeing is that I used a QEMU from riscv-to-apply.next from the maintainer tree [1] instead of master.
I was thinking about libvirt's master branch rather than QEMU's, but that makes sense. I'll check out v2 after building QEMU from the appropriate branch.
commit c13dfff2ded9cf2098a07e2426ffcad674602030 Author: Daniel Henrique Barboza <dbarboza@ventanamicro.com> Date: Tue Apr 11 15:35:10 2023 -0300
target/riscv: add query-cpy-definitions support
Since this hasn't made its way into master yet, maybe you can still get the maintainer to s/cpy/cpu/ for you? :)
Nice catch! I'll let him know. Thanks! Daniel
participants (2)
-
Andrea Bolognani -
Daniel Henrique Barboza