Re: GSoC 2022 CI project idea proposal

On Fri, Feb 25, 2022 at 11:31:39AM +0100, Erik Skultety wrote: > On Thu, Feb 24, 2022 at 06:49:29AM -0800, Andrea Bolognani wrote: > > I'm not entirely convinced that requiring the use of lcitool for this > > task is necessarily the best idea though. Ideally, it should be > > possible for developers to reproduce and debug CI failures locally > > without having to clone a second repository. It's fine to require > > lcitool for tasks that are performed by maintainers, but casual > > contributors should find all they need in libvirt.git itself. > > Unless they also want to test with VMs as well in which case lcitool is highly > recommended. The thing is that right now ci/helper wraps the Makefile. Let's > say you (or someone else) replaces it with a python equivalent, I don't think > that the number of contributors suddenly starting to use it will differ much > compared to the situation right now, but maybe I'm wrong :). > My motivation to add the support to lcitool is simple - right now there's an > RFC on adopting some infrastructure in order to run TCK-based functional tests > as part of GitLab. That's just the first stage, ultimately we want developers > to be able to run the same test suite locally without the need to wait for > gitlab and since the current design of the above is using nested virt, it is an > achievable goal. There was also a suggestion to adopt the Avocado framework for > a new test framework which would be largely based on TCK [1]. > Now, let's get to the point I'm trying to make :) - my expectation/vision is > that both the remote and local functional test executions will revolve around > lcitool and so eventually with the adoption of *some* framework it is going to > become a requirement that libvirt developers do perform functional testing > before submitting a patch series. With that said, it makes more sense in my > mind to focus on adding the container runtime support to lcitool in order to > move towards the bigger goal and have everything consolidated in a single > place. Alright, that's a pretty solid argument. And even if we ultimately decide that we don't want to require using lcitool, once we have a project-agnostic Python implementation of this logic it would be reasonably straightforward to turn it into a standalone tool similar to the current ci/helper, so we have an exit strategy. > > Another thing that has been bothering me is that neither 'ci/helper > > build' nor 'lcitool build' will actually perform the exact same build > > steps as the corresponding CI job, making them only mildly effective > > as debugging tools for CI failures. And of course each of these build > > recipes is maintained separately, so we have three similar but not > > quite identical scripts floating around. > > Yes, but on its own, I think that is a problem which can be solved separately. Agreed that the two tasks are not necessarily depending on each other, but there's significant overlap. I think it makes sense to rationalize how the build steps for a project are defined and maintained as part of adding "build in a local container" support to lcitool. Do you have any high-level concerns about the ci/build approach I vaguely described? The finer details are of course far from being set in stone, but I think the overall idea is solid and we should aim for it being implemented as we evolve our CI tooling.