Re: [libvirt PATCH 00/20] ci: Move GitLab build recipes to a standalone script

On Mon, Feb 06, 2023 at 03:45:12PM +0000, Daniel P. Berrangé wrote: > On Mon, Feb 06, 2023 at 02:52:57PM +0100, Erik Skultety wrote: > > This is a follow up to: > > https://listman.redhat.com/archives/libvir-list/2023-January/237201.html > > > > The effort here is to unify the way builds/tests are executed in GitLab CI vs > > local container executions and make another step forward in terms of > > reproducibility of (specifically) GitLab environments. > > > > Even though code to run all but one (coverity) jobs from GitLab via the > > build.sh script is added with this series, local behavior remains the same as > > before this series. The reason for that is that that will require more patches > > ridding of the Makefile which is currently used and instead integrate usage of > > lcitool with the ci/helper Python script which is currently the entry point for > > local container executions. > > snip > > > .gitlab-ci.yml | 56 ++++++++++------------- > > ci/Makefile | 16 ++++--- > > ci/build.sh | 121 +++++++++++++++++++++++++++++++++++++++++++------ > > ci/helper | 21 ++++++--- > > 4 files changed, 155 insertions(+), 59 deletions(-) > > mode change 100644 => 100755 ci/build.sh > > I'm really super unenthusiastic about this change, and also the similar > change to add an ci/integration.sh. Shell scripting is something we > worked hard to eliminate from libvirt. It is an awful language > to do anything non-trivial with, error handling, lack of data > structures, variable handling, portability are all bug generators. > > I know the .gitlab-ci.yml 'script' commands are technically shell > script, but they are pretty trivial bits and don't have to worry > about portability for dash vs bash etc, or complex control logic. > The majority of it is just a simple list of commands to invoke, > with the occasional conditional. > > The build.sh script is by contrast significantly more complex. By > the very nature of taking "N" separate gitlab jobs and multiplexing > them onto the one shell script, we're now having todo command line > arg parsing in shell and de-multiplexing back to commands. The CI > logic is now split up across even more sources - the gitlab config, > the build.sh script and the meson.build files, which I think is > worse for maintaining this too. True, except that even despite that, and I know what I'm talking about since I wrote the integration jobs templates, GitLab is super picky; you can't debug the syntax problems properly; certain common syntactic sugars don't work and have to be replaced by less common counterparts; using YAML for bash formatting leads to many table-flipping moments; you have to wait for Nx10 minutes before it fails with some ridiculous error you would have caught early locally but you couldn't because of the stage and artifact dependencies. So, once you have it finally in place it's unarguably nice, but the road towards that isn't really rainbow and unicorns either, it's still shell after all. Since I'm mainly working on this part of libvirt, I myself would appreciate if I could easily just run a single script instead of copy pasting commands one-by-one to a local VM to catch stupid errors quickly rather than wait for GitLab to fail. > > I appreciate the goal of being able to run CI commands locally, but > I'm not feeling this approach is a net win. I'd much rather just > having developers invoke the meson command directly, which is not > that hard. Well, the big picture here is about making running integration tests locally less painful than it is now...plus increase the amount of automation involved. That's the major driver here - libvirt needs to be built locally in a safe environment before integration tests could be run against it without touching the host. So, with this script, while I agree with everything you said about Bash, was just one step towards getting the automation I mentioned in place. I also considered Python (yeah..why), but that would be super akward. TL;DR: if we don't do anything about how we currently maintain the build recipes (note we're maintaining 2 already), everybody will continue ignoring the integration test suite, unless we enable merge requests where the status quo would be somewhat (but not completely) eliminated. With integration tests you can't ignore the aspect of getting feedback early compared to waiting for GitLab CI. > > If we do really want to provide something that 100% mirrors the > gitlab CI job commands though, I'm even more convinced now that > we should be using the .gitlab-ci.yml as the canonical source. > > Since I last mentioned that idea, I found out that this should > be something we can achieve via the gitlab runner 'exec' command. Haven't heard of ^this one, but I wonder how we can get something meaningful out of it for the purposes I mentioned above. Erik