Re: [libvirt PATCH 4/4] gitlab-ci: Introduce a new test 'integration' pipeline stage
On Wed, Mar 02, 2022 at 04:42:21PM +0100, Jiri Denemark wrote:
On Wed, Mar 02, 2022 at 09:43:24 +0000, Daniel P. Berrangé wrote:
> On Wed, Mar 02, 2022 at 08:42:30AM +0100, Erik Skultety wrote:
> > ...
> Ultimately when we switch to using merge requests, the integration tests
> should be run as a gating job, triggered from the merge train when the
> code gets applied to git, so that we prevent regressions actually making
> it into git master at all.
>
> Post-merge integration testing always exhibits the problem that people will
> consider it somebody else's problem to fix the regression. So effectively
> whoever creates the integration testing system ends up burdened with the
> job of investigating failures and finding someone to poke to fix it. With
> it run pre-merge then whoever wants to get their code merged needs to
> investigate the problems. Now sometimes the problems with of course be
> with the integration test system itself, not the submitters code, but
> this is OK because it leads to situation where the job of maintaining
> the integration tests are more equitably spread across all involved and
> builds a mindset that functional / integration testing is a critical
> part of delivering code, which is something we've lacked for too long
> in libvirt.
This is all fine as long as there's a way to explicitly merge something
even if CI fails. I don't like waiting for a fix in something completely
unrelated. For example, a MR with translations or an important bug fix
I haven't studied GitLab enough to rule it out straight away, but I'd like to
have some more intelligent hook (or whatnot) in place that could detect what
type of change is proposed so that documentation changes and translations would
not go through the whole pipeline and instead would be merged almost
immediately (after passing e.g. the docs build).
An important bug fix however must go through the whole pipeline IMO whether we
like it or not, that's the point, we need to ensure that we're not introducing
a regression to the best of our capabilities. As for CI infrastructure issues,
well, unfortunately that is an inevitable part of doing any automated tasks. By
introducing it upstream it should therefore become a collective responsibility,
so that if a CI issue occurs we need to work together to resolve it ASAP rather
than going around it and pushing a fix just because it takes long to execute.
Erik
should not be blocked by a bug in CI or an infrastructure issue. So
having a way (restricted, of course) to merge even if pipeline failed
would solve this issue.
Jirka