Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

On 12/14/21 01:41, Qu Wenruo wrote: > > > On 2021/12/14 00:49, Marc Zyngier wrote: >> On Mon, 13 Dec 2021 16:06:14 +0000, >> Peter Maydell <peter.maydell(a)linaro.org&gt; wrote: >>> >>> KVM on big.little setups is a kernel-level question really; I've >>> cc'd the kvmarm list. >> >> Thanks Peter for throwing us under the big-little bus! ;-) >> >>> >>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs(a)gmx.com&gt; wrote: >>>> >>>> >>>> >>>> On 2021/12/13 21:17, Michal Prívozník wrote: >>>>> On 12/11/21 02:58, Qu Wenruo wrote: >>>>>> Hi, >>>>>> >>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI >>>>>> CM4, >>>>>> with upstream kernels. >>>>>> >>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its >>>>>> little.BIG >>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it >>>>>> brings quite some troubles for VMs. >>>>>> >>>>>> In short, without proper cpuset to bind the VM to either all A72 >>>>>> cores >>>>>> or all A55 cores, the VM will mostly fail to boot. >> >> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the >> though of it makes me sick). >> >>>>>> >>>>>> Currently the working xml is: >>>>>> >>>>>>     <vcpu placement='static' cpuset='4-5'>2</vcpu> >>>>>>     <cpu mode='host-passthrough' check='none'/> >>>>>> >>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM >>>>>> will >>>>>> mostly fail to start up due to vcpu initialization failed with >>>>>> -EINVAL. >> >> Disclaimer: I know nothing about libvirt (and no, I don't want to >> know! ;-). >> >> However, for things to be reliable, you need to taskset the whole QEMU >> process to the CPU type you intend to use. > > Yep, that's what I'm doing. > >> That's because, AFAICT, >> QEMU will snapshot the system registers outside of the vcpu threads, >> and attempt to use the result to configure the actual vcpu threads. If >> they happen to run on different CPU types, the sysregs will differ in >> incompatible ways and an error will be returned. This may or may not >> be a bug, I don't know (I see it as a feature). > > Then this brings another question. > > If we can pin each vCPU to each physical core (both little and big), > then as long as the registers are per-vCPU based, it should be able to > pass both big and little cores to the VM. > > Yeah, I totally understand this screw up the scheduling, but that's at > least what (some insane) users want (just like me). > >> >> If you are annoyed with this behaviour, you can always use a different >> VMM that won't care about such difference (crosvm or kvmtool, to name >> a few). > > Sounds pretty interesting, a new world but without libvirt... > >> However, the guest will be able to observe the migration from >> one cpu type to another. This may or may not affect your guest's >> behaviour. > > Not sure if it's possible to pin each vCPU thread to each core, but let > me try. > Sure it is, for instance: <cputune> <vcpupin vcpu="0" cpuset="1-4,^2"/> <vcpupin vcpu="1" cpuset="0,1"/> <vcpupin vcpu="2" cpuset="2,3"/> <vcpupin vcpu="3" cpuset="0,4"/> <emulatorpin cpuset="1-3"/> <iothreadpin iothread="1" cpuset="5,6"/> <iothreadpin iothread="2" cpuset="7,8"/> </cputune>

pins vCPU#0 onto host CPUs 1-4, excluding 2; vCPU#1 onto host CPUs 0-1 and so on. You can also pin emulator (QEMU) and its iothreads. It's documented here: https://libvirt.org/formatdomain.html#cpu-tuning Michal