On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: > I feel like we should have separate deprecation entries for the > i686 host support, and for qemu-system-i386 emulator binary, as > although they're related they are independant features with > differing impact. eg removing qemu-system-i386 affects all > host architectures, not merely 32-bit x86 host, so I think we > can explain the impact more clearly if we separate them. Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is a superset. Removing support for building on 32 bit systems seems like a pity - it's one of a small number of ways to run 64 bit binaries on 32 bit systems, and the maintainance overhead is quite small.

On Mon, Feb 27, 2023 at 10:21:14AM -1000, Richard Henderson wrote: > On 2/27/23 10:12, Michael S. Tsirkin wrote: >> On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: >>> I feel like we should have separate deprecation entries for the >>> i686 host support, and for qemu-system-i386 emulator binary, as >>> although they're related they are independant features with >>> differing impact. eg removing qemu-system-i386 affects all >>> host architectures, not merely 32-bit x86 host, so I think we >>> can explain the impact more clearly if we separate them. >> >> Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is >> a superset. >> >> Removing support for building on 32 bit systems seems like a pity - it's >> one of a small number of ways to run 64 bit binaries on 32 bit systems, >> and the maintainance overhead is quite small. > > It's not that small. It only works for single-threaded system mode. It > silently does not honor atomicity for user-only mode, which is perhaps worse > not working at all.

We should probably block multi-threading on 32 bit then.

On Mon, Feb 27, 2023 at 10:21:14AM -1000, Richard Henderson wrote: >> Removing support for building on 32 bit systems seems like a pity - it's >> one of a small number of ways to run 64 bit binaries on 32 bit systems, >> and the maintainance overhead is quite small. > > It's not that small. Meaning there are lots of ways to run 64 bit binaries on 32 bit systems?

On 27/2/23 21:12, Michael S. Tsirkin wrote: > On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: >> I feel like we should have separate deprecation entries for the >> i686 host support, and for qemu-system-i386 emulator binary, as >> although they're related they are independant features with >> differing impact. eg removing qemu-system-i386 affects all >> host architectures, not merely 32-bit x86 host, so I think we >> can explain the impact more clearly if we separate them. > > Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is > a superset. Doesn't qemu-system-i386 start the CPU in a different mode that qemu-system-x86_64? Last time we discussed it, we mention adding -32 and -64 CLI flags to maintain compat, and IIRC this flag would add boot code to switch the CPU in 32-b. But then maybe I misunderstood. Thomas said, "CPUs must start in the same mode they start in HW".

On 27/02/2023 21.12, Michael S. Tsirkin wrote: > On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: > > I feel like we should have separate deprecation entries for the > > i686 host support, and for qemu-system-i386 emulator binary, as > > although they're related they are independant features with > > differing impact. eg removing qemu-system-i386 affects all > > host architectures, not merely 32-bit x86 host, so I think we > > can explain the impact more clearly if we separate them. > > Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is > a superset. > > Removing support for building on 32 bit systems seems like a pity - it's > one of a small number of ways to run 64 bit binaries on 32 bit systems, > and the maintainance overhead is quite small. Note: We're talking about 32-bit *x86* hosts here. Do you really think that someone is still using QEMU usermode emulation to run 64-bit binaries on a 32-bit x86 host?? ... If so, I'd be very surprised!

> In fact, keeping this support around forces correct use of > posix APIs such as e.g. PRIx64 which makes the code base > more future-proof. If you're concerned about PRIx64 and friends: We still continue to do compile testing with 32-bit MIPS cross-compilers and Windows 32-bit cross-compilers for now. The only thing we'd lose is the 32-bit "make check" run in the CI. Thomas

On Tue, Feb 28, 2023 at 03:19:20AM -0500, Michael S. Tsirkin wrote: > On Tue, Feb 28, 2023 at 08:49:09AM +0100, Thomas Huth wrote: > > On 27/02/2023 21.12, Michael S. Tsirkin wrote: > > > On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: > > > > I feel like we should have separate deprecation entries for the > > > > i686 host support, and for qemu-system-i386 emulator binary, as > > > > although they're related they are independant features with > > > > differing impact. eg removing qemu-system-i386 affects all > > > > host architectures, not merely 32-bit x86 host, so I think we > > > > can explain the impact more clearly if we separate them. > > > > > > Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is > > > a superset. > > > > > > Removing support for building on 32 bit systems seems like a pity - it's > > > one of a small number of ways to run 64 bit binaries on 32 bit systems, > > > and the maintainance overhead is quite small. > > > > Note: We're talking about 32-bit *x86* hosts here. Do you really think that > > someone is still using QEMU usermode emulation > > to run 64-bit binaries on a 32-bit x86 host?? ... If so, I'd be very surprised! > > I don't know - why x86 specifically? One can build a 32 bit binary on any host. > I think 32 bit x86 environments are just more common in the cloud. Can you point to anything that backs up that assertion. Clouds I've seen always give you a 64-bit environment, and many OS no longer even ship 32-bit installable media.

I would be surprised if 32-bit is above very very low single digits usage compared to x86_64.

> > > In fact, keeping this support around forces correct use of > > > posix APIs such as e.g. PRIx64 which makes the code base > > > more future-proof. > > > > If you're concerned about PRIx64 and friends: We still continue to do > > compile testing with 32-bit MIPS cross-compilers and Windows 32-bit > > cross-compilers for now. The only thing we'd lose is the 32-bit "make check" > > run in the CI. > > > > Thomas > > Yes - fundamentally 32 bit does not seem that different from e.g. > windows builds - we presumably support these but AFAIK CI does not > test these. We do compile test windows in CI via mingw, and we also do build and unit tests via msys. Even Windows has dropped 32-bit support though, and so the only reason we keep 32-bit Windows around is because of Windows 10. Once a Windows 12 comes along, we'll not need to support 32-bit Windows either. With regards, Daniel

-- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 28/02/2023 10.03, Michael S. Tsirkin wrote: > On Tue, Feb 28, 2023 at 08:59:52AM +0000, Daniel P. Berrangé wrote: > > On Tue, Feb 28, 2023 at 03:19:20AM -0500, Michael S. Tsirkin wrote: > > > On Tue, Feb 28, 2023 at 08:49:09AM +0100, Thomas Huth wrote: > > > > On 27/02/2023 21.12, Michael S. Tsirkin wrote: > > > > > On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: > > > > > > I feel like we should have separate deprecation entries for the > > > > > > i686 host support, and for qemu-system-i386 emulator binary, as > > > > > > although they're related they are independant features with > > > > > > differing impact. eg removing qemu-system-i386 affects all > > > > > > host architectures, not merely 32-bit x86 host, so I think we > > > > > > can explain the impact more clearly if we separate them. > > > > > > > > > > Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is > > > > > a superset. > > > > > > > > > > Removing support for building on 32 bit systems seems like a pity - it's > > > > > one of a small number of ways to run 64 bit binaries on 32 bit systems, > > > > > and the maintainance overhead is quite small. > > > > > > > > Note: We're talking about 32-bit *x86* hosts here. Do you really think that > > > > someone is still using QEMU usermode emulation > > > > to run 64-bit binaries on a 32-bit x86 host?? ... If so, I'd be very surprised! > > > > > > I don't know - why x86 specifically? One can build a 32 bit binary on any host. > > > I think 32 bit x86 environments are just more common in the cloud. > > > > Can you point to anything that backs up that assertion. Clouds I've > > seen always give you a 64-bit environment, and many OS no longer > > even ship 32-bit installable media. > > Sorry about being unclear. I meant that it seems easier to run CI in the > cloud in a 32 bit x64 environment than get a 32 bit ARM environment. It's still doable ... but for how much longer? We're currently depending on Fedora, but they also slowly drop more and more support for this environment, see e.g.:

https://www.theregister.com/2022/03/10/fedora_inches_closer_to_dropping/

On Tue, Feb 28, 2023 at 09:40:49AM +0000, Daniel P. Berrangé wrote: > On Tue, Feb 28, 2023 at 10:14:52AM +0100, Thomas Huth wrote: > > On 28/02/2023 10.03, Michael S. Tsirkin wrote: > > > On Tue, Feb 28, 2023 at 08:59:52AM +0000, Daniel P. Berrangé wrote: > > > > On Tue, Feb 28, 2023 at 03:19:20AM -0500, Michael S. Tsirkin wrote: > > > > > On Tue, Feb 28, 2023 at 08:49:09AM +0100, Thomas Huth wrote: > > > > > > On 27/02/2023 21.12, Michael S. Tsirkin wrote: > > > > > > > On Mon, Feb 27, 2023 at 11:50:07AM +0000, Daniel P. Berrangé wrote: > > > > > > > > I feel like we should have separate deprecation entries for the > > > > > > > > i686 host support, and for qemu-system-i386 emulator binary, as > > > > > > > > although they're related they are independant features with > > > > > > > > differing impact. eg removing qemu-system-i386 affects all > > > > > > > > host architectures, not merely 32-bit x86 host, so I think we > > > > > > > > can explain the impact more clearly if we separate them. > > > > > > > > > > > > > > Removing qemu-system-i386 seems ok to me - I think qemu-system-x86_64 is > > > > > > > a superset. > > > > > > > > > > > > > > Removing support for building on 32 bit systems seems like a pity - it's > > > > > > > one of a small number of ways to run 64 bit binaries on 32 bit systems, > > > > > > > and the maintainance overhead is quite small. > > > > > > > > > > > > Note: We're talking about 32-bit *x86* hosts here. Do you really think that > > > > > > someone is still using QEMU usermode emulation > > > > > > to run 64-bit binaries on a 32-bit x86 host?? ... If so, I'd be very surprised! > > > > > > > > > > I don't know - why x86 specifically? One can build a 32 bit binary on any host. > > > > > I think 32 bit x86 environments are just more common in the cloud. > > > > > > > > Can you point to anything that backs up that assertion. Clouds I've > > > > seen always give you a 64-bit environment, and many OS no longer > > > > even ship 32-bit installable media. > > > > > > Sorry about being unclear. I meant that it seems easier to run CI in the > > > cloud in a 32 bit x64 environment than get a 32 bit ARM environment. > > > > It's still doable ... but for how much longer? We're currently depending on > > Fedora, but they also slowly drop more and more support for this > > environment, see e.g.: > > FWIW, we should cull our fedora-i386-cross.docker dockerfile and > replace it with a debian i686 dockerfile generated by lcitool. > There's no compelling reason why i686 should be different from > all our other cross builds which are based on Debian. The Debian > lcitool generated container would have access to a wider range > of deps than our hand written Fedora one. > > > https://www.theregister.com/2022/03/10/fedora_inches_closer_to_dropping/ > > With regards, > Daniel ... and is closer to where 32 bit is likely to be deployed which is systems like e.g. raspberry pi os which until recently was only 32 bit.

On Tue, Feb 28, 2023 at 11:39:39AM +0100, Markus Armbruster wrote: > The question to answer: Is 32 bit x86 worth its upkeep? Two > sub-questions: 1. Is it worth the human attention? 2. Is it worth > (scarce!) CI minutes? 3. Is it worth arguing about?

On Tue, Feb 28, 2023 at 12:12:22PM +0100, Thomas Huth wrote: > On 28/02/2023 11.51, Michael S. Tsirkin wrote: > > On Tue, Feb 28, 2023 at 11:39:39AM +0100, Markus Armbruster wrote: > > > The question to answer: Is 32 bit x86 worth its upkeep? Two > > > sub-questions: 1. Is it worth the human attention? 2. Is it worth > > > (scarce!) CI minutes? > > > > 3. Is it worth arguing about? > > You are aware of the problems we're currently struggeling with, aren't you? > Darn, we're having *SEVERE* problems with the CI, the QEMU project ran out > of CI minutes for the second time this year, and you ask whether it's worth > arguing about??? You're not serious with this question, are you? > > Thomas Yah just couldn't resist. How many minutes do we use per month btw?

If it's not worth arguing, then we merge Thomas's patch.

We're struggling quite badly with our CI minutes on the shared gitlab runners, so we urgently need to think of ways to cut down our supported build and target environments. qemu-system-i386 and qemu-system-arm are not really required anymore, since nobody uses KVM on the corresponding systems for production anymore, and the -x86_64 and -arch64 variants are a proper superset of those binaries. So it's time to deprecate them and the corresponding 32-bit host environments now. This is a follow-up patch series from the previous discussion here: https://lore.kernel.org/qemu-devel/20230130114428.1297295-1-thuth@redhat.... where people still mentioned that there is still interest in certain support for 32-bit host hardware. But as far as I could see, there is no real need for 32-bit host support for system emulation on x86 and arm anymore, so it should be fine if we drop these host environments now (these are also the two architectures that contribute the most to the long test times in our CI, so we would benefit a lot by dropping those).

On 27/02/2023 19.38, Daniel P. Berrangé wrote: > On Mon, Feb 27, 2023 at 12:10:48PM +0100, Thomas Huth wrote: > > We're struggling quite badly with our CI minutes on the shared > > gitlab runners, so we urgently need to think of ways to cut down > > our supported build and target environments. qemu-system-i386 and > > qemu-system-arm are not really required anymore, since nobody uses > > KVM on the corresponding systems for production anymore, and the > > -x86_64 and -arch64 variants are a proper superset of those binaries. > > So it's time to deprecate them and the corresponding 32-bit host > > environments now. > > > > This is a follow-up patch series from the previous discussion here: > > > > https://lore.kernel.org/qemu-devel/20230130114428.1297295-1-thuth@redhat.... > > > > where people still mentioned that there is still interest in certain > > support for 32-bit host hardware. But as far as I could see, there is > > no real need for 32-bit host support for system emulation on x86 and > > arm anymore, so it should be fine if we drop these host environments > > now (these are also the two architectures that contribute the most to > > the long test times in our CI, so we would benefit a lot by dropping > > those). > > Your description here is a little ambiguous about what's being > proposed. When you say dropping 32-bit host support do you mean > just for the system emulator binaries, or for QEMU entirely ? Just for system emulation. Some people said that user emulation still might be useful for some 32-bit environments. > And when the deprecation period is passed, are you proposing > to actively prevent 32-bit builds, or merely stopping CI testing > and leave 32-bit builds still working if people want them ? CI is the main pain point, so that's the most important thing. So whether we throw a warning or a hard error while configuring the build, I don't care too much.

On Tue, Feb 28, 2023 at 08:39:49AM +0100, Thomas Huth wrote: > On 27/02/2023 19.38, Daniel P. Berrangé wrote: >> On Mon, Feb 27, 2023 at 12:10:48PM +0100, Thomas Huth wrote: >>> We're struggling quite badly with our CI minutes on the shared >>> gitlab runners, so we urgently need to think of ways to cut down >>> our supported build and target environments. qemu-system-i386 and >>> qemu-system-arm are not really required anymore, since nobody uses >>> KVM on the corresponding systems for production anymore, and the >>> -x86_64 and -arch64 variants are a proper superset of those binaries. >>> So it's time to deprecate them and the corresponding 32-bit host >>> environments now. >>> >>> This is a follow-up patch series from the previous discussion here: >>> >>> https://lore.kernel.org/qemu-devel/20230130114428.1297295-1-thuth@redhat.... >>> >>> where people still mentioned that there is still interest in certain >>> support for 32-bit host hardware. But as far as I could see, there is >>> no real need for 32-bit host support for system emulation on x86 and >>> arm anymore, so it should be fine if we drop these host environments >>> now (these are also the two architectures that contribute the most to >>> the long test times in our CI, so we would benefit a lot by dropping >>> those). >> >> Your description here is a little ambiguous about what's being >> proposed. When you say dropping 32-bit host support do you mean >> just for the system emulator binaries, or for QEMU entirely ? > > Just for system emulation. Some people said that user emulation still might > be useful for some 32-bit environments. > >> And when the deprecation period is passed, are you proposing >> to actively prevent 32-bit builds, or merely stopping CI testing >> and leave 32-bit builds still working if people want them ? > > CI is the main pain point, so that's the most important thing. So whether we > throw a warning or a hard error while configuring the build, I don't care > too much. If we're merely wanting to drop CI support, we can do that any time and deprecation is not required/expected. We should only be using deprecation where we're explicitly intending that the code will cease to work.

On Tue, Feb 28, 2023 at 09:05:16PM +0100, Thomas Huth wrote: > Well, without CI, I assume that the code will bitrot quite fast (considering > that there are continuous improvements to TCG, for example). We have lots of hosts which we don't test with CI. They don't bitrot because people do testing before release.

"Michael S. Tsirkin" <mst(a)redhat.com&gt; writes: > On Tue, Feb 28, 2023 at 09:05:16PM +0100, Thomas Huth wrote: >> Well, without CI, I assume that the code will bitrot quite fast (considering >> that there are continuous improvements to TCG, for example). > We have lots of hosts which we don't test with CI. They don't bitrot > because people do testing before release. This is what RCs are for. > We did releases before CI - it is a cost/benefit thing. Dropping 32-bit x86 from CI feels like a no-brainer in the current situation. As to deprecating 32-bit x86: the people by far most qualified to judge the "cost/benefit thing" are the regulars who are bearing the cost, i.e. the people who are actually maintaining it. Their opinion should overrule any "but somebody out there might still want to use it". Maintainers, please state your opinion, if any: aye or nay. Richard tells us "the maint overhead is large." Makes me think he's in favour of dropping 32-bit x86. Richard? Peter seems to be reluctant to drop 32-bit ARM at this point. Peter?