Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

On Mon, 13 Dec 2021 16:06:14 +0000, Peter Maydell <peter.maydell(a)linaro.org&gt; wrote: > > KVM on big.little setups is a kernel-level question really; I've > cc'd the kvmarm list. Thanks Peter for throwing us under the big-little bus! ;-) > > On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs(a)gmx.com&gt; wrote: >> >> >> >> On 2021/12/13 21:17, Michal Prívozník wrote: >>> On 12/11/21 02:58, Qu Wenruo wrote: >>>> Hi, >>>> >>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, >>>> with upstream kernels. >>>> >>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG >>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it >>>> brings quite some troubles for VMs. >>>> >>>> In short, without proper cpuset to bind the VM to either all A72 cores >>>> or all A55 cores, the VM will mostly fail to boot. s/A55/A53/. There were thankfully no A72+A55 ever produced (just the though of it makes me sick). >>>> >>>> Currently the working xml is: >>>> >>>> <vcpu placement='static' cpuset='4-5'>2</vcpu> >>>> <cpu mode='host-passthrough' check='none'/> >>>> >>>> But even with vcpupin, pinning each vcpu to each physical core, VM will >>>> mostly fail to start up due to vcpu initialization failed with -EINVAL. Disclaimer: I know nothing about libvirt (and no, I don't want to know! ;-). However, for things to be reliable, you need to taskset the whole QEMU process to the CPU type you intend to use. That's because, AFAICT, QEMU will snapshot the system registers outside of the vcpu threads, and attempt to use the result to configure the actual vcpu threads. If they happen to run on different CPU types, the sysregs will differ in incompatible ways and an error will be returned. This may or may not be a bug, I don't know (I see it as a feature).