Re: [PATCH v2 4/4] virtio-net: Add support for USO features

On 2024/07/30 2:00, Peter Xu wrote: > On Mon, Jul 29, 2024 at 04:58:03PM +0100, Daniel P. Berrangé wrote: > > On Fri, Jul 26, 2024 at 04:47:40PM -0400, Peter Xu wrote: > > > On Fri, Jul 26, 2024 at 04:17:12PM +0100, Daniel P. Berrangé wrote: > > > > > > > > In terms of launching QEMU I'd imagine: > > > > > > > > $QEMU -machine pc-q35-9.1 -platform linux-6.9 ...args... > > > > > > > > Any virtual machine HW features which are tied to host kernel features > > > > would have their defaults set based on the requested -platform. The > > > > -machine will be fully invariant wrt the host kernel. > > > > > > > > You would have -platform hlep to list available platforms, and > > > > corresonding QMP "query-platforms" command to list what platforms > > > > are supported on a given host OS. > > > > > > > > Downstream distros can provide their own platforms definitions > > > > (eg "linux-rhel-9.5") if they have kernels whose feature set > > > > diverges from upstream due to backports. > > > > > > > > Mgmt apps won't need to be taught about every single little QEMU > > > > setting whose default is derived from the kernel. Individual > > > > defaults are opaque and controlled by the requested platform. > > > > > > > > Live migration has clearly defined semantics, and mgmt app can > > > > use query-platforms to validate two hosts are compatible. > > > > > > > > Omitting -platform should pick the very latest platform that is > > > > cmpatible with the current host (not neccessarily the latest > > > > platform built-in to QEMU). > > > > > > This seems to add one more layer to maintain, and so far I don't know > > > whether it's a must. > > > > > > To put it simple, can we simply rely on qemu cmdline as "the guest ABI"? I > > > thought it was mostly the case already, except some extremely rare > > > outliers. > > > > > > When we have one host that boots up a VM using: > > > > > > $QEMU1 $cmdline > > > > > > Then another host boots up: > > > > > > $QEMU2 $cmdline -incoming XXX > > > > > > Then migration should succeed if $cmdline is exactly the same, and the VM > > > can boot up all fine without errors on both sides. > > > > > > AFAICT this has nothing to do with what kernel is underneath, even not > > > Linux? I think either QEMU1 / QEMU2 has the option to fail. But if it > > > didn't, I thought the ABI should be guaranteed. > > > > We've got two mutually conflicting goals with the machine type > > definitions. > > > > Primarily we use them to ensure stable ABI, but an important > > secondary goal is to enable new tunables to have new defaults > > set, without having to update every mgmt app. The latter > > works very well when the defaults have no dependancy on the > > platform kernel/OS, but breaks migration when they do have a > > platform dependancy. > > > > > - Firstly, never quietly flipping any bit that affects the ABI... > > > > > > - Have a default value of off, then QEMU will always allow the VM to boot > > > by default, while advanced users can opt-in on new features. We can't > > > make this ON by default otherwise some VMs can already fail to boot, > > > > > > - If the host doesn't support the feature while the cmdline enabled it, > > > it needs to fail QEMU boot rather than flipping, so that it says "hey, > > > this host does not support running such VM specified, due to XXX > > > feature missing". > > > > > > That's the only way an user could understand what happened, and IMHO that's > > > a clean way that we stick with QEMU cmdline on defining the guest ABI, > > > while in which the machine type is the fundation of such definition, as the > > > machine type can decides many of the rest compat properties. And that's > > > the whole point of the compat properties too (to make sure the guest ABI is > > > stable). > > > > > > If kernel breaks it easily, all compat property things that we maintain can > > > already stop making sense in general, because it didn't define the whole > > > guest ABI.. > > > > > > So AFAIU that's really what we used for years, I hope I didn't overlook > > > somehting. And maybe we don't yet need the "-platform" layer if we can > > > keep up with this rule? > > > > We've failed at this for years wrt enabling use of new defaults that have > > a platform depedancy, so historical practice isn't a good reference. > > > > There are 100's (possibly 1000's) of tunables set implicitly as part of > > the machine type, and of those, libvirt likely only exposes a few 10's > > of tunables. The vast majority are low level details that no mgmt app > > wants to know about, they just want to accept QEMU's new defaults, > > while preserving machine ABI. This is a good thing. No one wants the > > burden of wiring up every single tunable into libvirt and mgmt apps. > > > > This is what the "-platform" concept would be intended to preserve. It > > would allow a way to enable groups of settings that have a platform level > > dependancy, without ever having to teach either libvirt or the mgmt apps > > about the individual tunables. > > Do you think we can achieve similar goal by simply turning the feature to > ON only after a few QEMU releases? I also mentioned that idea below. > > https://lore.kernel.org/r/ZqQNKZ9_OPhDq2AK@x1n > > So far it really sounds like the right thing to do to me to fix all similar > issues, even without introducing anything new we need to maintain. > > To put that again, what we need to do is this: > > - To start: we should NEVER turn any guest ABI relevant bits > automatically by QEMU, for sure.. > > - When introducing any new device feature that may both (1) affects guest > ABI, and (2) depends on host kernel features, we set those default > values to OFF always at start. So this already covers old machine > types, no compat property needed so far. > > - We always fail hard on QEMU boot whenever we detected such property is > not supported by the current host when with ON (and since it's OFF by > default it must be that the user specified that ON). > > - When after a stablized period of time for that new feature to land most > kernels (we may consider to look at how major Linux distros updates the > kernel versions) when we're pretty sure the new feature should be > available on most of the QEMU modern users, we add a patch to make the > property default ON on the new machine type, add a compat property for > old machines. > > The last bullet also means we'll start to fail new machine type from > booting when running that very new QEMU on a very old kernel, but that's > the trade-off, and when doing it right on "stablizing the feature in the > kernel world", it should really be corner case. The user should simply > invoke an old machine type on that old kernel, even if the qemu is new. docs/about/build-platforms.rst already defines supported platforms. One of the supported platforms is Debian 11 (bullseye), and it carries Linux 5.10, which was released December 2020. If we follow this platform support, a new feature added to upstream Linux may take about 4 years before it gets enabled by default on QEMU. As an upstream developer, I feel it is too long, but I'm sure there are different opinions from different perspectives.