On Tue, Feb 22, 2022 at 05:04:02PM +0100, Erik Skultety wrote: I've been meaning to replace the make-based logic used for spawning local containers with a Python equivalent and get rid of ci/Makefile entirely. There was some progress last year, but I got sidetracked and never managed to finish the job. So obviously I'm in favor of doing more work in that area, especially if it's someone else doing it O:-) I'm not entirely convinced that requiring the use of lcitool for this task is necessarily the best idea though. Ideally, it should be possible for developers to reproduce and debug CI failures locally without having to clone a second repository. It's fine to require lcitool for tasks that are performed by maintainers, but casual contributors should find all they need in libvirt.git itself. Another thing that has been bothering me is that neither 'ci/helper build' nor 'lcitool build' will actually perform the exact same build steps as the corresponding CI job, making them only mildly effective as debugging tools for CI failures. And of course each of these build recipes is maintained separately, so we have three similar but not quite identical scripts floating around. Here's my (fairly hand-wavy :) idea of how things should work. Each project's repository should contain its build steps in the form of a ci/build script. The exact calling interface for this script will have to be determined, but based on existing usage at the very least we need to be able to specify the build type (regular, cross and with dependencies built from git) and target-specific settings (for example whether to skip building RPMs). So I can imagine the invocation looking something like $ TYPE=native RPMS=skip ci/build The build steps that currently live in the .native_build_job, .cross_build_job and .native_git_build_job templates in .gitlab-ci.yml will all be moved to the ci/build script. It shouldn't be necessary to alter them significantly in the process. With this interface defined, we can change 'lcitool manifest' so that the jobs it generates go from x86_64-almalinux-8-clang: extends: .native_build_job needs: - x86_64-almalinux-8-container variables: CC: clang NAME: almalinux-8 RPM: skip to x86_64-almalinux-8-clang: needs: - x86_64-almalinux-8-container script: - TYPE=native NAME=almalinux-8 CC=clang RPM=skip ci/build Of course it would still be possible to tell lcitool to use a custom template for the job instead, which might be necessary for setting up CI caching and the like. For simple cases though, you'd be able to use the default implementation. 'lcitool build' would also be changed so that it invokes the ci/build script in the project's repository. The last missing piece would then be to finish converting the ci/helper script that exists in libvirt to Python and make it so calling that entry point also ultimately in running the ci/build script. Of course having multiple copies of the ci/helper script and its logic for listing / running containers around is problematic because they would get out of sync over time... Perhaps we can maintain that script as part of libvirt-ci.git, and generate a local copy to be stored in each project's repository at the same time as other CI-related files such as Dockerfiles are generated? That sounds like it could work, but admittedly this is the fuzziest part of the entire plan :) -- Andrea Bolognani / Red Hat / Virtualization

On Fri, Feb 25, 2022 at 11:31:39AM +0100, Erik Skultety wrote: Alright, that's a pretty solid argument. And even if we ultimately decide that we don't want to require using lcitool, once we have a project-agnostic Python implementation of this logic it would be reasonably straightforward to turn it into a standalone tool similar to the current ci/helper, so we have an exit strategy. Agreed that the two tasks are not necessarily depending on each other, but there's significant overlap. I think it makes sense to rationalize how the build steps for a project are defined and maintained as part of adding "build in a local container" support to lcitool. Do you have any high-level concerns about the ci/build approach I vaguely described? The finer details are of course far from being set in stone, but I think the overall idea is solid and we should aim for it being implemented as we evolve our CI tooling. -- Andrea Bolognani / Red Hat / Virtualization

On Fri, Feb 25, 2022 at 12:30:08PM +0100, Erik Skultety wrote: Just to make sure we're on the same page, what do you mean by "copied to the Dockerfile"? The CI job can call the script directly from the local clone of the repository just like a developer would on their machine - no copying necessary. The Dockerfile describes the environment a build will happen in, not the build steps. We're stepping into premature bikeshedding territory here :) but I would expect this to look like 'lcitool build $target --container' or something along those lines. Making it possible to specify an alternate path to the build script could be a neat feature, but we should have a sensible default that makes it possible for people to simply not care most of the time. -- Andrea Bolognani / Red Hat / Virtualization

On Fri, Feb 25, 2022 at 05:49:39AM -0800, Andrea Bolognani wrote: I've never got on very well with the 'lcitool build' command as its a bit of a black box you don't have much control over. In its current impl, it also means that lcitool has to know about build commands for each project which is unfortunate. If we're going to wire up support for containers, I think we should start by just creating a 'lcitool run $target-or-host' command that brings up an environment with an interactive shell, with the current source present. eg $ lcitool run fedora-35 We could then let it take a command + args to run as extra args eg $ lcitool run fedora-35 meson build $ lcitool run fedora-35 ninja -C build would just work with VM, but for containers you would need the container FS to be persistent across runs. This could be achieved by giving the container an explicit name, so you just re-restart the same container each time instead of using a throwaway container. The basic idea though is that running stuff inside the container/VM is just the same as running stuff in your local checkout. The exact same command name(s0, just prefixed with 'lcitool run fedora-35'. Or in my case I'd add an alias alias f35='lcitool run fedora-35' so i can do $ f35 meson build $ f35 ninja -C bujld etc Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Feb 25, 2022 at 08:36:10AM -0800, Andrea Bolognani wrote: I guess that depends on your POV really, as that's not the way I prefer to debug things. I want to have an interactive shell and invokes the individual commands myself, so that I can have ability to customize their invokation to alter loggging or debugging settings, or choose different build options to understand their effect. IOW, what matters to me is the getting an environment up and running, with the source tree available, and ready to step through the build commands. A 'ci/build' shell script is ok to just reproduce a failure, but since it doesn't let me tweak things, I'm not likely to use it much for debugging. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|