Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

On 2021/12/14 00:49, Marc Zyngier wrote: > On Mon, 13 Dec 2021 16:06:14 +0000, > Peter Maydell <peter.maydell(a)linaro.org&gt; wrote: >> >> KVM on big.little setups is a kernel-level question really; I've >> cc'd the kvmarm list. > > Thanks Peter for throwing us under the big-little bus! ;-) > >> >> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs(a)gmx.com&gt; wrote: >>> >>> >>> >>> On 2021/12/13 21:17, Michal Prívozník wrote: >>>> On 12/11/21 02:58, Qu Wenruo wrote: >>>>> Hi, >>>>> >>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, >>>>> with upstream kernels. >>>>> >>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG >>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it >>>>> brings quite some troubles for VMs. >>>>> >>>>> In short, without proper cpuset to bind the VM to either all A72 cores >>>>> or all A55 cores, the VM will mostly fail to boot. > > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the > though of it makes me sick). > >>>>> >>>>> Currently the working xml is: >>>>> >>>>> <vcpu placement='static' cpuset='4-5'>2</vcpu> >>>>> <cpu mode='host-passthrough' check='none'/> >>>>> >>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will >>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL. > > Disclaimer: I know nothing about libvirt (and no, I don't want to > know! ;-). > > However, for things to be reliable, you need to taskset the whole QEMU > process to the CPU type you intend to use. Yep, that's what I'm doing.

> That's because, AFAICT, > QEMU will snapshot the system registers outside of the vcpu threads, > and attempt to use the result to configure the actual vcpu threads. If > they happen to run on different CPU types, the sysregs will differ in > incompatible ways and an error will be returned. This may or may not > be a bug, I don't know (I see it as a feature). Then this brings another question. If we can pin each vCPU to each physical core (both little and big), then as long as the registers are per-vCPU based, it should be able to pass both big and little cores to the VM.

Yeah, I totally understand this screw up the scheduling, but that's at least what (some insane) users want (just like me).

> > If you are annoyed with this behaviour, you can always use a different > VMM that won't care about such difference (crosvm or kvmtool, to name > a few). Sounds pretty interesting, a new world but without libvirt... > However, the guest will be able to observe the migration from > one cpu type to another. This may or may not affect your guest's > behaviour. Not sure if it's possible to pin each vCPU thread to each core, but let me try.

> I personally find the QEMU behaviour reasonable. KVM/arm64 make little > effort to support BL virtualisation as design choice (I value my > sanity), and userspace is still in control of the placement. > >>>>> This brings a problem, in theory RK3399 SoC should out-perform BCM2711 >>>>> in multi-core performance, but if a VM can only be bind to either A72 or >>>>> A55 cores, then the performance is no longer competitive against >>>>> BCM2711, wasting the PCIE 2.0 x4 capacity. > > Vote with your money. If you too think that BL systems are utter crap, > do not buy them! Or treat them as 'two systems in one', which is what > I do. From that angle, this is of great value! ;-) I guess I'm setting my expectation too high for rk3399, just seeing its multi-thread perf beating RPI4 and has better IO doesn't mean it's a perfect fit for VM.