On Fri, Jul 26, 2024 at 09:03:24AM +0200, Thomas Huth wrote: The later.

On 26/07/2024 09.25, Michael S. Tsirkin wrote: From my experience, it should rather be the former. We've seen similar issues with the s390x machine in the past when trying to automatically enable features depending on the availability of a kernel features. While it looks nicer at a very first glance ("hey, a new feature is available, we enable that for you, dear user!"), you end up in migration hell pretty quickly. Maybe we could elevate the "--nodefaults" command line switch to avoid enabling such features automatically? Anyway, while we're discussing solutions: We are in softfreeze already. Should we disable the UFO bits in the new 9.1 machine type for the time being to avoid that more people are running into this problem? Thomas

On Fri, Jul 26, 2024 at 07:39:46PM +0200, Thomas Huth wrote: At the moment I'm looking at solutions for 9.2 I don't see how we can do much for 9.1. I mean we can move it back to behave like 8.1 (IIRC), but that is not much. -- MST

On Fri, Jul 26, 2024 at 09:48:02AM +0100, Daniel P. Berrangé wrote: Any more elaboration on this idea? Would it be easily feasible in implementation? I'd second any sane solution that we can avoid happening similar breakages in the future. I also wonder what else might be easily affected like this too when migration can break with changed kernel or changed HW. I suppose the CPU model is well covered by Libvirt so we're fine at least on x86 etc. While IIUC KVM always have such thoughts in mind, so that KVM will make sure to not break an userspace in such way or it'll simply be a KVM bug and fixed. Thanks, -- Peter Xu

On Fri, Jul 26, 2024 at 04:17:12PM +0100, Daniel P. Berrangé wrote: This seems to add one more layer to maintain, and so far I don't know whether it's a must. To put it simple, can we simply rely on qemu cmdline as "the guest ABI"? I thought it was mostly the case already, except some extremely rare outliers. When we have one host that boots up a VM using: $QEMU1 $cmdline Then another host boots up: $QEMU2 $cmdline -incoming XXX Then migration should succeed if $cmdline is exactly the same, and the VM can boot up all fine without errors on both sides. AFAICT this has nothing to do with what kernel is underneath, even not Linux? I think either QEMU1 / QEMU2 has the option to fail. But if it didn't, I thought the ABI should be guaranteed. That's why I think this is a migration violation, as 99.99% of other device properties should be following this rule. The issue here is, we have the same virtio-net-pci cmdline on both sides in this case, but the ABI got break. That's also why I was suggesting if the property contributes to the guest ABI, then AFAIU QEMU needs to: - Firstly, never quietly flipping any bit that affects the ABI... - Have a default value of off, then QEMU will always allow the VM to boot by default, while advanced users can opt-in on new features. We can't make this ON by default otherwise some VMs can already fail to boot, - If the host doesn't support the feature while the cmdline enabled it, it needs to fail QEMU boot rather than flipping, so that it says "hey, this host does not support running such VM specified, due to XXX feature missing". That's the only way an user could understand what happened, and IMHO that's a clean way that we stick with QEMU cmdline on defining the guest ABI, while in which the machine type is the fundation of such definition, as the machine type can decides many of the rest compat properties. And that's the whole point of the compat properties too (to make sure the guest ABI is stable). If kernel breaks it easily, all compat property things that we maintain can already stop making sense in general, because it didn't define the whole guest ABI.. So AFAIU that's really what we used for years, I hope I didn't overlook somehting. And maybe we don't yet need the "-platform" layer if we can keep up with this rule? Thanks, -- Peter Xu

On Sun, Jul 28, 2024 at 11:19 PM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: I don't get why eBPF matters here. It is something that is not noticed by the guest and we have a fallback anyhow. I may miss something but I think "Auto" doesn't make sense to libvirt. Thanks

On Mon, Jul 29, 2024 at 01:45:12PM +0900, Akihiko Odaki wrote: Isn't keeping the default AUTO the same as before when it used to be ON? I mean, AUTO in a qemu cmdline doesn't guarantee guest API either. Indeed it looks like it's a step forward to make ON having the clear semantics of "fail when unsupported". It's just that I am not sure how useful is AUTO here, because anyway we'll need to break ON semantics even with AUTO, so that an old QEMU script with USO=ON used to boot on old kernels but not it won't. What I was trying to say is whether we should make the default parameter to be migratable. IOW, it looks to me AUTO should deserve a migration blocker when chosen. After all, Libvirt hopefully shouldn't use AUTO at all but only ON/OFF, while any user when not caring much on these perf details should always use OFF on any kernel dependent features that may affect the guest ABI. Thanks, -- Peter Xu

On Tue, Jul 30, 2024 at 12:43 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: It's a bug then. Qemu can fallback to tuntap if it sees issues in vhost. Thanks

On Tue, Jul 30, 2024 at 10:57 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: Such fallback is by design since the introduction of vhost. Yes but what I want to say is that eBPF RSS is different from those segmentation offloads. And technically, Qemu can do fallback for offloads (as RSC did). Thanks

On Tue, Jul 30, 2024 at 11:12 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: It should be a bug (and I remember we disabled vhost when the patches were merged). Have you tested it in a guest to see if it can see RSC when vhost is enabled? I suspect we need to add the RSC bit into current kernel_feature_bits: /* Features supported by host kernel. */ static const int kernel_feature_bits[] = { VIRTIO_F_NOTIFY_ON_EMPTY, VIRTIO_RING_F_INDIRECT_DESC, VIRTIO_RING_F_EVENT_IDX, VIRTIO_NET_F_MRG_RXBUF, VIRTIO_F_VERSION_1, VIRTIO_NET_F_MTU, VIRTIO_F_IOMMU_PLATFORM, VIRTIO_F_RING_PACKED, VIRTIO_F_RING_RESET, VIRTIO_NET_F_HASH_REPORT, VHOST_INVALID_FEATURE_BIT }; As RSC won't be provided by TUN/TAP anyhow. True, technically, vDPA can fallback to SVQ, but it's another topic. Thanks

On Tue, Jul 30, 2024 at 11:29 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: Yes. Down the road, we probably need to change the behaviour of disabling vhost-net. To make sure we are on the same page. I just want to point out that eBPF RSS is not a good example in this context. It works only for tuntap, so we should stick to the behaviour of trying to fallback to userspace if we can as we've already had a userspace fallback. This is the fundamental difference with other features (like segmentation offload) or backend (vDPA) that doesn't have an existing fallback. Thanks

@Akihiko Odaki The RSC is supported with vhost and without vhost The 'in-qemu RSC' is related to VIRTIO_NET_F_RSC_EXT feature, it is intended for one specific WHCK test only and should not be used in any functional setup. When it is used the vhost should be off On Tue, Jul 30, 2024 at 1:23 PM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: > > On 2024/07/30 12:45, Jason Wang wrote: > > On Tue, Jul 30, 2024 at 11:29 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: > >> > >> On 2024/07/30 12:17, Jason Wang wrote: > >>> On Tue, Jul 30, 2024 at 11:12 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: > >>>> > >>>> On 2024/07/30 12:03, Jason Wang wrote: > >>>>> On Tue, Jul 30, 2024 at 10:57 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: > >>>>>> > >>>>>> On 2024/07/30 11:04, Jason Wang wrote: > >>>>>>> On Tue, Jul 30, 2024 at 12:43 AM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: > >>>>>>>> > >>>>>>>> On 2024/07/29 23:29, Peter Xu wrote: > >>>>>>>>> On Mon, Jul 29, 2024 at 01:45:12PM +0900, Akihiko Odaki wrote: > >>>>>>>>>> On 2024/07/29 12:50, Jason Wang wrote: > >>>>>>>>>>> On Sun, Jul 28, 2024 at 11:19 PM Akihiko Odaki <akihiko.odaki(a)daynix.com&gt; wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On 2024/07/27 5:47, Peter Xu wrote: > >>>>>>>>>>>>> On Fri, Jul 26, 2024 at 04:17:12PM +0100, Daniel P. Berrangé wrote: > >>>>>>>>>>>>>> On Fri, Jul 26, 2024 at 10:43:42AM -0400, Peter Xu wrote: > >>>>>>>>>>>>>>> On Fri, Jul 26, 2024 at 09:48:02AM +0100, Daniel P. Berrangé wrote: > >>>>>>>>>>>>>>>> On Fri, Jul 26, 2024 at 09:03:24AM +0200, Thomas Huth wrote: > >>>>>>>>>>>>>>>>> On 26/07/2024 08.08, Michael S. Tsirkin wrote: > >>>>>>>>>>>>>>>>>> On Thu, Jul 25, 2024 at 06:18:20PM -0400, Peter Xu wrote: > >>>>>>>>>>>>>>>>>>> On Tue, Aug 01, 2023 at 01:31:48AM +0300, Yuri Benditovich wrote: > >>>>>>>>>>>>>>>>>>>> USO features of virtio-net device depend on kernel ability > >>>>>>>>>>>>>>>>>>>> to support them, for backward compatibility by default the > >>>>>>>>>>>>>>>>>>>> features are disabled on 8.0 and earlier. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Signed-off-by: Yuri Benditovich <yuri.benditovich(a)daynix.com&gt; > >>>>>>>>>>>>>>>>>>>> Signed-off-by: Andrew Melnychecnko <andrew(a)daynix.com&gt; > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Looks like this patch broke migration when the VM starts on a host that has > >>>>>>>>>>>>>>>>>>> USO supported, to another host that doesn't.. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> This was always the case with all offloads. The answer at the moment is, > >>>>>>>>>>>>>>>>>> don't do this. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> May I ask for my understanding: > >>>>>>>>>>>>>>>>> "don't do this" = don't automatically enable/disable virtio features in QEMU > >>>>>>>>>>>>>>>>> depending on host kernel features, or "don't do this" = don't try to migrate > >>>>>>>>>>>>>>>>> between machines that have different host kernel features? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Long term, we need to start exposing management APIs > >>>>>>>>>>>>>>>>>> to discover this, and management has to disable unsupported features. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Ack, this likely needs some treatments from the libvirt side, too. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> When QEMU automatically toggles machine type featuers based on host > >>>>>>>>>>>>>>>> kernel, relying on libvirt to then disable them again is impractical, > >>>>>>>>>>>>>>>> as we cannot assume that the libvirt people are using knows about > >>>>>>>>>>>>>>>> newly introduced features. Even if libvirt is updated to know about > >>>>>>>>>>>>>>>> it, people can easily be using a previous libvirt release. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> QEMU itself needs to make the machine types do that they are there > >>>>>>>>>>>>>>>> todo, which is to define a stable machine ABI. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> What QEMU is missing here is a "platform ABI" concept, to encode > >>>>>>>>>>>>>>>> sets of features which are tied to specific platform generations. > >>>>>>>>>>>>>>>> As long as we don't have that we'll keep having these broken > >>>>>>>>>>>>>>>> migration problems from machine types dynamically changing instead > >>>>>>>>>>>>>>>> of providing a stable guest ABI. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Any more elaboration on this idea? Would it be easily feasible in > >>>>>>>>>>>>>>> implementation? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In terms of launching QEMU I'd imagine: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> $QEMU -machine pc-q35-9.1 -platform linux-6.9 ...args... > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Any virtual machine HW features which are tied to host kernel features > >>>>>>>>>>>>>> would have their defaults set based on the requested -platform. The > >>>>>>>>>>>>>> -machine will be fully invariant wrt the host kernel. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> You would have -platform hlep to list available platforms, and > >>>>>>>>>>>>>> corresonding QMP "query-platforms" command to list what platforms > >>>>>>>>>>>>>> are supported on a given host OS. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Downstream distros can provide their own platforms definitions > >>>>>>>>>>>>>> (eg "linux-rhel-9.5") if they have kernels whose feature set > >>>>>>>>>>>>>> diverges from upstream due to backports. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Mgmt apps won't need to be taught about every single little QEMU > >>>>>>>>>>>>>> setting whose default is derived from the kernel. Individual > >>>>>>>>>>>>>> defaults are opaque and controlled by the requested platform. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Live migration has clearly defined semantics, and mgmt app can > >>>>>>>>>>>>>> use query-platforms to validate two hosts are compatible. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Omitting -platform should pick the very latest platform that is > >>>>>>>>>>>>>> cmpatible with the current host (not neccessarily the latest > >>>>>>>>>>>>>> platform built-in to QEMU). > >>>>>>>>>>>>> > >>>>>>>>>>>>> This seems to add one more layer to maintain, and so far I don't know > >>>>>>>>>>>>> whether it's a must. > >>>>>>>>>>>>> > >>>>>>>>>>>>> To put it simple, can we simply rely on qemu cmdline as "the guest ABI"? I > >>>>>>>>>>>>> thought it was mostly the case already, except some extremely rare > >>>>>>>>>>>>> outliers. > >>>>>>>>>>>>> > >>>>>>>>>>>>> When we have one host that boots up a VM using: > >>>>>>>>>>>>> > >>>>>>>>>>>>> $QEMU1 $cmdline > >>>>>>>>>>>>> > >>>>>>>>>>>>> Then another host boots up: > >>>>>>>>>>>>> > >>>>>>>>>>>>> $QEMU2 $cmdline -incoming XXX > >>>>>>>>>>>>> > >>>>>>>>>>>>> Then migration should succeed if $cmdline is exactly the same, and the VM > >>>>>>>>>>>>> can boot up all fine without errors on both sides. > >>>>>>>>>>>>> > >>>>>>>>>>>>> AFAICT this has nothing to do with what kernel is underneath, even not > >>>>>>>>>>>>> Linux? I think either QEMU1 / QEMU2 has the option to fail. But if it > >>>>>>>>>>>>> didn't, I thought the ABI should be guaranteed. > >>>>>>>>>>>>> > >>>>>>>>>>>>> That's why I think this is a migration violation, as 99.99% of other device > >>>>>>>>>>>>> properties should be following this rule. The issue here is, we have the > >>>>>>>>>>>>> same virtio-net-pci cmdline on both sides in this case, but the ABI got > >>>>>>>>>>>>> break. > >>>>>>>>>>>>> > >>>>>>>>>>>>> That's also why I was suggesting if the property contributes to the guest > >>>>>>>>>>>>> ABI, then AFAIU QEMU needs to: > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Firstly, never quietly flipping any bit that affects the ABI... > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Have a default value of off, then QEMU will always allow the VM to boot > >>>>>>>>>>>>> by default, while advanced users can opt-in on new features. We can't > >>>>>>>>>>>>> make this ON by default otherwise some VMs can already fail to boot, > >>>>>>>>>>>> > >>>>>>>>>>>> It may not be necessary the case that old features are supported by > >>>>>>>>>>>> every systems. In an extreme case, a user may migrate a VM from Linux to > >>>>>>>>>>>> Windows, which probably doesn't support any offloading at all. A more > >>>>>>>>>>>> convincing scenario is RSS offloading with eBPF; using eBPF requires a > >>>>>>>>>>>> privilege so we cannot assume it is always available even on the latest > >>>>>>>>>>>> version of Linux. > >>>>>>>>>>> > >>>>>>>>>>> I don't get why eBPF matters here. It is something that is not noticed > >>>>>>>>>>> by the guest and we have a fallback anyhow. > >>>>>>>> > >>>>>>>> It is noticeable for the guest, and the fallback is not effective with > >>>>>>>> vhost. > >>>>>>> > >>>>>>> It's a bug then. Qemu can fallback to tuntap if it sees issues in vhost. > >>>>>> > >>>>>> We can certainly fallback to in-QEMU RSS by disabling vhost, but I would > >>>>>> not say lack of such fallback is a bug. > >>>>> > >>>>> Such fallback is by design since the introduction of vhost. > >>>>> > >>>>>> We don't provide in-QEMU > >>>>>> fallback for other offloads. > >>>>> > >>>>> Yes but what I want to say is that eBPF RSS is different from those > >>>>> segmentation offloads. And technically, Qemu can do fallback for > >>>>> offloads (as RSC did). > >>>> > >>>> Well, I couldn't find any code disabling vhost for the in-QEMU RSC > >>>> implementation. > >>> > >>> It should be a bug (and I remember we disabled vhost when the patches > >>> were merged). Have you tested it in a guest to see if it can see RSC > >>> when vhost is enabled? > >>> > >>> I suspect we need to add the RSC bit into current kernel_feature_bits: > >>> > >>> /* Features supported by host kernel. */ > >>> static const int kernel_feature_bits[] = { > >>> VIRTIO_F_NOTIFY_ON_EMPTY, > >>> VIRTIO_RING_F_INDIRECT_DESC, > >>> VIRTIO_RING_F_EVENT_IDX, > >>> VIRTIO_NET_F_MRG_RXBUF, > >>> VIRTIO_F_VERSION_1, > >>> VIRTIO_NET_F_MTU, > >>> VIRTIO_F_IOMMU_PLATFORM, > >>> VIRTIO_F_RING_PACKED, > >>> VIRTIO_F_RING_RESET, > >>> VIRTIO_NET_F_HASH_REPORT, > >>> VHOST_INVALID_FEATURE_BIT > >>> }; > >>> > >>> As RSC won't be provided by TUN/TAP anyhow. > >> > >> Adding the RSC bit does not let QEMU disable vhost for RSC, but instead > >> it implicitly disables RSC in my understanding. > > > > Yes. > > > >> It is still better than > >> advertising the availability of that feature while it is missing. > > > > Down the road, we probably need to change the behaviour of disabling vhost-net. > > > >> > >>> > >>>> > >>>> Looking at the code, I also found the case of vhost-vdpa. vhost can be > >>>> simply disabled if it is backed by tuntap, but it is not the case for vDPA. > >>> > >>> True, technically, vDPA can fallback to SVQ, but it's another topic. > >> > >> My point of this discussion is that we cannot enable features just > >> because they are sufficiently old or because the user claims QEMU runs > >> on Linux sufficiently new. eBPF requires privilege, and vDPA requires > >> hardware feature. A fallback is not a silver bullet either, and there > >> are situations that providing a fallback is not a trivial task. > > > > To make sure we are on the same page. I just want to point out that > > eBPF RSS is not a good example in this context. > > > > It works only for tuntap, so we should stick to the behaviour of > > trying to fallback to userspace if we can as we've already had a > > userspace fallback. This is the fundamental difference with other > > features (like segmentation offload) or backend (vDPA) that doesn't > > have an existing fallback. > > Some (probably not all) offloads are implemented in hw/net/net_tx_pkt.c. > They are not wired up to behave as a fallback when tuntap's vhost is > enabled as the in-QEMU RSS is not. In either case, we need to pay some > effort to wiring things. > > I'm not sure it is worthwhile. I think there is a high chance that > selectively disabling vhost and keeping RSS enabled with fallback will > result in worse performance than keeping vhost enabled and disabling > RSS. Such a fallback can still function as an emergency escape hatch, > but it is also incomplete as we don't have fallbacks for other features. > I would rather make any features missing in the vhost backend fail to > keep things consistent. > > Regards, > Akihiko Odaki

On Fri, Jul 26, 2024 at 04:47:40PM -0400, Peter Xu wrote: We've got two mutually conflicting goals with the machine type definitions. Primarily we use them to ensure stable ABI, but an important secondary goal is to enable new tunables to have new defaults set, without having to update every mgmt app. The latter works very well when the defaults have no dependancy on the platform kernel/OS, but breaks migration when they do have a platform dependancy. We've failed at this for years wrt enabling use of new defaults that have a platform depedancy, so historical practice isn't a good reference. There are 100's (possibly 1000's) of tunables set implicitly as part of the machine type, and of those, libvirt likely only exposes a few 10's of tunables. The vast majority are low level details that no mgmt app wants to know about, they just want to accept QEMU's new defaults, while preserving machine ABI. This is a good thing. No one wants the burden of wiring up every single tunable into libvirt and mgmt apps. This is what the "-platform" concept would be intended to preserve. It would allow a way to enable groups of settings that have a platform level dependancy, without ever having to teach either libvirt or the mgmt apps about the individual tunables. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 2024/07/30 2:00, Peter Xu wrote: docs/about/build-platforms.rst already defines supported platforms. One of the supported platforms is Debian 11 (bullseye), and it carries Linux 5.10, which was released December 2020. If we follow this platform support, a new feature added to upstream Linux may take about 4 years before it gets enabled by default on QEMU. As an upstream developer, I feel it is too long, but I'm sure there are different opinions from different perspectives. Regards, Akihiko Odaki

On Mon, Jul 29, 2024 at 01:00:30PM -0400, Peter Xu wrote: Turning a feature with a platform dependency to "on" implies that the machine type will cease to work out of the box for platforms which lack the feature. IMHO that's not acceptable behaviour for any of our supported platforms. IOW, "after a few QEMU releases" implies a delay of as much as 5 years, while we wait for platforms which don't support the feature to drop out of our supported targets list. I don't think that'll satisfy the desire to get the new feature available to users as soon as practical for their particular platform. Our supported platform list determines when this will be, and given our current criteria, this can be as long as 5 years. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Tue, Jul 30, 2024 at 02:13:51PM -0400, Peter Xu wrote: If it is left to libvirt, then it would very likely end up being a user opt-in, not auto-enabled. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Tue, Jul 30, 2024 at 03:11:03PM -0400, Peter Xu wrote: This is not what we did historically. Why should we start now? -- MST

On Tue, Jul 30, 2024 at 04:03:53PM -0400, Peter Xu wrote: Randomly is an overstatement. You need to switch between kernels where this feature differs. We did it with a ton of features in the past, donnu why we single out USO now. Basically downstreams just don't separately add kernel features vs qemu features. There's little reason for them to do so. -- MST

On Wed, Jul 31, 2024 at 5:33 AM Michael S. Tsirkin <mst(a)redhat.com&gt; wrote: I guess the reason is that for offload features other than USO they are landed in early kernels so we don't have a chance to test/meet this case. But this is not the case for USO. Thanks

On Wed, Jul 31, 2024 at 08:04:24AM +0100, Daniel P. Berrangé wrote: Pushing it to domain xml does not really help, migration will still fail unexpectedly (after wasting a ton of resources copying memory, and getting a downtime bump, I might add). The right solution is to have a tool that can query backends, and that given the results from all of the cluster, generate a set of parameters that will ensure migration works. Kind of like qemu-img, but for migration. -- MST

On Wed, Jul 31, 2024 at 8:58 PM Peter Xu <peterx(a)redhat.com&gt; wrote: This seems to be very hard for vhost-users. Probably, letting Qemu to probe is much easier than rewriting the probe in the upper layer. Thanks

On Wed, Jul 31, 2024 at 08:57:52AM -0400, Peter Xu wrote: Agreed that it's a lot of work. However, this means we can never add new features without also teaching libvirt to enable them. How about we add some kind of command on source qemu to return description of all working features? Then when qemu is started on destination, this data can be passed in, and validated. Hmm? -- MST

On Thu, Aug 01, 2024 at 01:51:00AM -0400, Michael S. Tsirkin wrote: Something like that could work indeed. I'm thinking whether it shouldn't require a new QMP command; that sounds more work, and we also needs Libvirt cooperations so QEMU migration will still fail. I wonder whether we can integrate it into migration handshake that I referred previously in our TODO item here: https://wiki.qemu.org/ToDo/LiveMigration#Migration_handshake The "device handshake" part (in the previous plan) was that we at least can verify VMSD fields matching on both sides - VMSDs are defined in both QEMU binaries, so migration can do that already without device opt-in. What we can do on top of that (or even, before that) is, maybe, allow device to opt-in in such handshake besides an "VMSD check", so that there can be something hooked to the VMSDs or similar structures, so the src QEMU's device A can talk to dest QEMU's device A making sure everything is good for migration. Virtio can handshake on host feature lists and we can fail the whole handshake there. Same to -cpu, or vDPA, as long as opt-in hook is provided on both sides. The good side of it is it sounds natural to integrate this with a handshake (when we can have it). Meanwhile, we restrict everything within the device scope, so neither QEMU nor migration needs to know what happened exactly. Would that sound workable and better? Besides, I also wonder what's our next step for this issue. Should we fix this on the safe side, and only set ON by default when we have the handshake ready (in whatever form, either above, or a new QMP command)? It's just that the handshake in general may still need some thoughts, so I'm not sure how fast that can ready, considering our very limited bandwidth so far. Maybe that can be done separately, but I remember Dan used to suggest we do handshake right in one shot, and I tend to agree that'll be nicer. Thanks, -- Peter Xu

On Thu, Aug 01, 2024 at 01:51:00AM -0400, Michael S. Tsirkin wrote: This feels like only half a solution, and not the most helpful half. It prevents you accidentally migrating to a host that lacks some features, but doesn't help with starting a VM that has migrate compatible features in the first place. From a user POV, the latter is what's most important. Checking for incompatible features is just a safety net that you should never need to hit, if QEMU was configured suitably to start with. So to ensure a QEMU is started with migration compatible features will still require teaching libvirt about every single feature that has a host kernel dependancy, so libvirt (or the app using libvirt) knows to turn this off. This is alot more work for both libvirt & the mgmt app, than having QEMU provide the generic "platforms" concept which is extensible without needing further work outside QEMU. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Aug 01, 2024 at 11:50:40AM -0400, Michael S. Tsirkin wrote: We don't have to support all possible combinations IMHO. If a user really does require precise control over every combination of some settings, then exposing those tunables in libvirt is inevitable. The platform concept only has to be able to express a "good enough" subset of combinations, such that it is unlikely users will need to have fine tuning for most of the tunables. We might end up exposing a handful of tunables in libvirt anyway, but as long as we get the common case satisifed, we'll eliminate most of the ongoing burden. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Aug 01, 2024 at 02:05:54PM +0900, Akihiko Odaki wrote: I'm not sure if I read it right. Perhaps you meant something more generic than -platform but similar? For example, "-profile [PROFILE]" qemu cmdline, where PROFILE can be either "perf" or "compat", while by default to "compat"? If so, I think I get the idea, but it'll be challenging in at least these aspects: - We already have (at least.. that I'm aware of) three layers of specifying a property for a device, they are: (1) default value (2) compat property (from machine type definitions) (3) qemu cmdline (specify one property explicitly) So far, there's an order we apply these (1-3), while (3) has the top priority to overwrite (1-2), and (2) to overwrite (1). The new "-profile", if I read it right, introduce (4), and it's already unclear to me how that interacts with (3) when -profile says "turn FEAT1 on" while cmdline says otherwise. It can make things very compilcated, IMHO. - This still will break the "QEMU cmdline defines the guest ABI", e.g., consider this USO* thing that we boot an old machine type on a new system that has QEMU+Linux USO* all enabled. We specify "-profile perf" there. Then when we try to migrate to another older QEMU it'll still fail the migration instead of any way telling us "migration is not compatible". So even if it helps the user turning on knobs, it doesn't sound like to fix the problem we're working on? For whatever profile setup, it sounds like more applicable to a Libvirt option that user can choose. That may avoid above two concerns I have, especially the latter. But I really don't know much on Libvirt, and this can be some extra effort too on top of either QEMU / Libvirt, and we may need to justify worthwhile. Do we really concern about users not enabling features that much? I thought users always can manually change the XML and add whatever they need, and device properties do not like too special here to me. I mean, we have bunch of "features" exported as new "-devices" and users must opt-in for them by changing the XML. We never worried on user not using them. I doubt whether we worried too much on user not opt-in, especially for performance features, because they're, IMHO, targeting advanced users. Thanks, -- Peter Xu

On Thu, Aug 01, 2024 at 11:15:47AM -0400, Michael S. Tsirkin wrote: With the -platform concept, libvirt wouldn't need to know anything about the settings being used, nor the defaults. Consider how it works for machine types. Libvirt queries the machine types, and gets a list back, and QEMU expresses a default. eg saying that 'pc-i440fx-9.1.0' is aliased to 'pc'. So libvirt can expand 'pc' to a particular version that QEMU has chosen as the default. Conceptually I could see something similar working for the -platform concept. Libvirt would ask QEMU for all the "platform" variants that are available on the current running kernel. QEMU can reply with the list, and indicate which of those is the "newest" in some manner. Absent any preference from the mgmt app, libvirt would use whichever one QEMU indicates was the newest. This optimizes for best featureset on the current kernel, as the cost of possibly reduced migration compatibility. When a mgmt app is caring about migration, they would explicitly tell libvirt which platform version to use, just as they would explicitly ask for a specific machine type version, rather than accepting the 'pc' default. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Aug 02, 2024 at 01:30:51PM +0900, Akihiko Odaki wrote: In practice, lots of people would benefit from ability to migrate using host features (checking that hosts are compatibile, as they often are). If we are going to go to great lengths adding new interfaces, I think that would be a really useful thing to address. -- MST

On 2024/08/03 0:05, Peter Xu wrote: Do you mean merging platform dumps should be done with another command? I think we will want to know the QOM tree is in use when implementing -merge-platform. For example, you cannot define a "platform" when e.g., you don't know what netdev backend (e.g., user, vhost-net, vhost-vdpa) is connected to virtio-net devices. Of course we can include those information in dumps, but we don't do so for VMState. Perhaps it may be named no-cross-migrate or something. There are lots of details we need to figure out. There is certainly downside and upside to add a migration blocker and I don't have a strong opinion here. Regards, Akihiko Odaki

On Fri, Aug 02, 2024 at 12:26:22PM -0400, Peter Xu wrote: We have two big issues around migration compatibility we never solved: - some guest visible behaviour depends on a package outside of qemu: as that package can change, so can qemu behaviour - sometimes we change guest visible behaviour and only discover this after the release: fixing that breaks migration to one version, not fixing breaks migration to another These, to me, look similar enough that I feel we should look at them together from QAPI POV. Both issues sometimes can have work-arounds, enabling these would be nice. Also, both issues have a clean solution, which can come in two flavors: 1. basic: detecting incompatibility and not starting qemu on destination (or failing migration, possibly early, which I consider a less clean solution). 2. advanced: ability to go from a set of configurations to a flag making them compatible. -- MST

On 2024/08/03 1:26, Peter Xu wrote: I think we will still need the QOM tree in that case. I think the platform information will look somewhat similar to VMState, which requires the QOM tree to interpret. I suggest disabling all offload features of virtio-net with 9.2. I want to keep things consistent so I want to disable all at once. This change will be very uncomfortable for us, who are implementing offload features, but I hope it will motivate us to implement a proper solution. That said, it will be surely a breaking change so we should wait for 9.1 before making such a change. By the way, I am wondering perhaps the "no-cross-migrate" scenario can be implemented relatively easy in a way similar to compatibility properties. The idea is to add the "no-cross-migrate" property to machines. If the property is set to "on", all offload features of virtio-net will be set to "auto". virtio-net will then probe the offload features and enable available offloading features. Regards, Akihiko Odaki

On Sun, Aug 04, 2024 at 09:08:05AM -0400, Peter Xu wrote: This is why I think we (also?) need a way to dump config and then give it to qemu on destination. To have a simple way to management to know whether migration has a chance of working.

On Mon, Aug 05, 2024 at 04:27:43PM +0900, Akihiko Odaki wrote: I see. I am not familiar with the status quo there, so I'll leave that to you and other experts that know better on this.. Personally I do care more on Linux, as that's what we ship within RH.. Ah ok. But note that there's still a slight difference: we need to avoid AUTO being an option, at all, IMHO. It's about making qemu cmdline the ABI: when with AUTO it's still possible the user uses AUTO on both sides, then ABI may not be guaranteed. AUTO would be fine if: (1) the property doesn't affect guest ABI, or (2) the AUTO bit will always generate the same thing on both hosts. However USO* isn't such case.. so the AUTO option is IMHO not wanted. What I mentioned above "x-virtio-net-offload-enforce" shouldn't add anything new to "uso"; it still can only be ON/OFF. However it should affect "flip that to OFF automatically" or "fail the boot" behavior on missing features. Basically you're saying that no-cross-migrate has lower priority than specific feature bits. That's OK to me. Thanks, -- Peter Xu

On Thu, Aug 08, 2024 at 08:43:22PM +0900, Akihiko Odaki wrote: Hmm yes; I wished we can have device properties that user can never specify, but only set from internals. It's just that applying a compat property so far require a generic device property. Or say, it'll be nice that compat property can tweak a class variable too then no property to introduce. We could even add a migration blocker for x-virtio-net-offload-enforce=ON, but again it could be too aggresive. I think it might be better we bet nobody will even know there's the parameter, so it won't be used in manual setup. OTOH, "guest_uso4" can be too easiy to find there's the AUTO option: I normally use ",guest_uso4=?" to just dump the possible values. Thanks, -- Peter Xu

On 2024/08/08 22:55, Peter Xu wrote: We can detect and reject AUTO when cross-migrate=on if desired, but I'm not sure it's worthwhile. Regards, Akihiko Odaki

On 2024/08/05 16:30, Michael S. Tsirkin wrote: An obvious alternative is to set cross-migrate=off by default (I dropped the no- prefix because no-cross-migrate=off is confusing). I don't have a particular idea whether cross-migrate should be on or off by default. This is a trade-off of safety and performance. In general, I believe safety should come first before performance. On the other hand, disabling offload features is a breaking change. QEMU also has -only-migratable option; it is more consistent to make the additional assurance for migration opt-in instead of opt-out. Finally, I see migration across hosts as an advanced feature, and perhaps it can be justified to make it more like an optional feature. Regards, Akihiko Odaki

On 2024/08/05 17:23, Michael S. Tsirkin wrote: "cross-migrate" means migration among hosts with different platforms (e.g., different kernels, vDPA devices). If cross-migrate=off, QEMU can still migrate on the same host (checkpoint and restart). QEMU can also migrate across hosts if the user ensures they are on the same platform.

On 2024/08/05 19:08, Michael S. Tsirkin wrote: Downgrades will not work with cross-migrate=off. Users who want downgrades should use cross-migrate=on.

On 2024/08/06 22:29, Michael S. Tsirkin wrote: We can do nothing in such a case. Whether it is on a single host or multiple hosts, we cannot support migration if features once enabled disappear. Regards, Akihiko Odaki

On 2024/08/08 19:54, Michael S. Tsirkin wrote: Whethere it is checkpoint/restart or any other migration, I expect platform features won't disappear from the host(s); we can't readily support migration in such a situation. When platform features won't disappear, for checkpoint/restart, we can enable all available features without disrupting migration; cross-migrate=off will instruct that. However, if we are migrating a VM across hosts and the user doesn't ensure they are on the same platform, we cannot enable platform features even if we are sure that platform features already present on a host won't disappear because some hosts may not have features in the first place. We can set cross-migrate=on in such a case to disable optional platform features. Regards, Akihiko Odaki

On 2024/08/08 20:12, Michael S. Tsirkin wrote: I think you meant cross-migrate=off, which is the current behavior. I am not suggesting forcing cross-migrate=on or even making it default. I have shown four possible scenarios earlier[a]: 1) Migration everywhere 2) Migration on specific machines 3) Migration on some known platforms 4) No migration (migration on nowhere) Taking the discussion with Peter, I amend 4) as follows: 4*) Migration on one platform (checkpoint/restore) cross-migrate=on is a complete solution for 1). 2) is dealt with another proposal of mine.[b] 3) can be solved with the -platform proposal by Daniel.[c] 4*) is what QEMU currently implements. [a] https://lore.kernel.org/all/39a8bb8b-4191-4f41-aaf7-06df24bf3280@daynix.com/ [b] https://lore.kernel.org/all/2da4ebcd-2058-49c3-a4ec-8e60536e5cbb@daynix.com/ [c] https://lore.kernel.org/all/ZqO7cR-UiGpX2rk0@redhat.com/ Regards, Akihiko Odaki

On Thu, Aug 08, 2024 at 07:12:14AM -0400, Michael S. Tsirkin wrote: IMHO this is totally different thing. It's not about proposing a new feature yet so far, it's about how we should fix a breakage first. And that's why I think we should fix it even in the simple way first, then we consider anything more benefitial from perf side without breaking anything, which should be on top of that. Thanks, -- Peter Xu

On Thu, Aug 08, 2024 at 10:47:28AM -0400, Michael S. Tsirkin wrote: IMHO it's not a hack. It's a proper fix to me to disable it by default for now. OTOH, having it ON always even knowing it can break migration is a hack to me, when we don't have anything else to guard the migration. Exactly mixing kernel versions will be tricky to users to identify, but that's, AFAICT, exactly happening everywhere. We can't urge user to always use the exact same kernels when we're talking about a VM cluster. That's why I think allowing migration to work across those kernels matter. I will agree there's no rush iff RHEL9 kernel won't backport TAP at all, otherwise this will trigger between y-stream after people upgrades partial of the clusters. Thanks, -- Peter Xu

On 2024/08/09 21:50, Fabiano Rosas wrote: I have wrote a bit about the expectation on the platform earlier[1], but let me summarize it here. 1. I expect the user will not downgrade the platform of hosts after setting up a VM. This is essential to enable any platform feature. 2. The user is allowed to upgrade the platform of hosts gradually. This results in a situation with mixed platforms. The oldest platform is still not older than the platform the VM is set up for. This enables the gradual deployment strategy. 3. the user is allowed to downgrade the platform of hosts to the version used when setting up the VM. This enables rollbacks in case of regression. With these expectations, we can ensure migratability by a) enabling platform features available on all hosts when setting up the VM and b) saving the enabled features. This is covered with my -dump-platform/-merge-platform/-use-platform proposal[2]. Regards, Akihiko Odaki [1] https://lore.kernel.org/r/2b62780c-a6cb-4262-beb5-81d54c14f545@daynix.com [2] https://lore.kernel.org/all/2da4ebcd-2058-49c3-a4ec-8e60536e5cbb@daynix.com/

On 2024/08/18 16:03, Michael S. Tsirkin wrote: No, but I want to move "[PATCH v3 0/5] virtio-net: Convert feature properties to OnOffAuto" forward: https://patchew.org/QEMU/20240714-auto-v3-0-e27401aabab3@daynix.com/ This will clarify the existence of the "auto" semantics, which is to enable a platform feature based on availability. [2] will be regarded as a feature to improve the handling of the "auto" semantics once this change lands. Regards, Akihiko Odaki > >> Regards, >> Akihiko Odaki >> >> [1] >> https://lore.kernel.org/r/2b62780c-a6cb-4262-beb5-81d54c14f545@daynix.com >> [2] >> https://lore.kernel.org/all/2da4ebcd-2058-49c3-a4ec-8e60536e5cbb@daynix.com/ >

On 2024/08/09 0:25, Peter Xu wrote: I think neither of them is a hack; they just deal with different scenarios summarized in [1]. We need apply a solution appropriate for each scenario, or we will end up with a broken system. Regards, Akihiko Odaki [1] https://lore.kernel.org/r/770300ac-7ed3-4aba-addb-b3f987cc6376@daynix.com/

On Tue, Jul 30, 2024 at 02:02:27AM +0900, Akihiko Odaki wrote: I think for starters, we can just have dump-features as a QEMU command. Pass it on command line on destination. Achieves the same thing as making userspace pass each flag manually, but without the pain of teaching management to enable each new feature. -- MST