Re: [PATCH] ci: Install python RPMs from libvirt/libvirt-python CI artifacts

On Thu, Oct 13, 2022 at 09:43:50AM +0200, Erik Skultety wrote: > On Mon, Oct 10, 2022 at 11:09:51AM +0100, Daniel P. Berrangé wrote: > > On Mon, Oct 10, 2022 at 11:46:43AM +0200, Peter Krempa wrote: > > > After addition of the new libvirt-client-qemu sub-package which is using > > > python bindings (thus creating a circular dependency between the libvirt > > > and libvirt-python projects) the integration jobs fail with: > > > > > > Error: > > > Problem: conflicting requests > > > - nothing provides python3-libvirt >= 8.9.0-1.el9 needed by libvirt-client-qemu-8.9.0-1.el9.x86_64 > > > > Hmm, I wonder, should we really be trying to install *all* RPMs ? > > > > IIUC, we're only running integration tests for QEMU, so it ought > > to be sufficient to do only have the QEMU driver and virsh client > > and Perl APIs. eg can we change ci/integrate-templates.yml to > > do > > > > createrepo libvirt-rpms > > createrepo libvirt-perl > > dnf install libvirt-daemon-qemu libvirt-client perl-Sys-Virt > > Sorry for the delay, I was actually integrating ^this suggestion and GitLab is, > well, slow to test these changes with. Anyhow, I vote for Peter's original > patch as it's the simplest and cleanest way to fix the problem, why? Here's > what I ended up with on my branch: > > diff --git a/ci/integration-template.yml b/ci/integration-template.yml > index d4799b2ca8..9c58a1b71c 100644 > --- a/ci/integration-template.yml > +++ b/ci/integration-template.yml > @@ -25,7 +25,31 @@ > > > .install-deps: &install-deps > - - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* > + - sudo dnf install -y createrepo > + - createrepo libvirt-rpms > + - createrepo libvirt-perl-rpms > + - |- > + for repo in libvirt-rpms libvirt-perl-rpms; do > + cat << EOF >> "$SCRATCH_DIR/ci.repo" > + [CI-$repo] > + name=ci-$repo > + baseurl=file://$(pwd)/$repo > + enabled=1 > + gpgcheck=0 > + EOF > + done > + - sudo mv "$SCRATCH_DIR/ci.repo" /etc/yum.repos.d/ > + - sudo dnf install -y > + libvirt-daemon-config-nwfilter > + libvirt-daemon-config-network > + libvirt-daemon-driver-interface > + libvirt-daemon-driver-network > + libvirt-daemon-driver-nodedev > + libvirt-daemon-driver-nwfilter > + libvirt-daemon-driver-qemu > + libvirt-daemon-driver-secret > + libvirt-daemon-driver-storage > + perl-Sys-Virt > - sudo pip3 install --prefix=/usr avocado-framework FYI, can be simplified to: - sudo dnf install -y createrepo - createrepo libvirt-rpms - createrepo libvirt-perl-rpms - dnf config-manager --add-repo libvirt-rpms - dnf config-manager --add-repo libvirt-perl-rpms

- sudo dnf install -y libvirt-daemon-config-nwfilter libvirt-daemon-config-network libvirt-daemon-qemu perl-Sys-Virt > Not only does it require more steps, is ugly because of the heredoc combined > with YAML's explicit line break preservation (|-), libvirt-daemon-driver-qemu > also doesn't pull all the needed bits as it was suggested, e.g. > libvirt-daemon-config-nwfilter which is needed by the tests. Although the tests > don't need virtinterfaced, I still had to include it because the logic which > restarts all the daemons would have to be adjusted as well; and then one day we > add a test for virtinterfaced we'd have to add it back, so IMO installing and > restarting all the daemons is the cleanest and most transparent way of setting > up the CI env even though we're not making use of all of it now (and so I ended > up naming all the secondary drivers explicitly). NB, I suggested libvirt-daemon-qemu, not libvirt-daemon-driver-qemu, because the former pulls in all the secondary driver packages, so would only miss the config packages.

> The other huge inconvenience with the proposed approach compared to Peter's is > debugging the failures. What I currently do with any integration CI failures is > that I connect to the runner machine and spin up an instance of one of the > images from which gitlab provisions the environment, download the RPM artifacts > from libvirt and libvirt-perl, install and then debug - again, why? Because > unfortunately trying to do this locally is still less convenient even though > we've done plenty of work in lcitool to improve the status quo. So, back to > your suggestion - I'd have to create the repositories manually, install a few > explicit packages and then test; sorry, that maybe fine for the CI, but doesn't > beat the following in any way: > > wget <libvirt-rpms-uri> -O libvirt_rpms.zip > wget <libvirt-perl-rpms-uri> -O libvirt_perl_rpms.zip > wget <libvirt-python-rpms-uri> -O libvirt_python_rpms.zip > unzip <yada yada> > > dnf install libvirt_rpms/* libvirt_perl_rpms/* libvirt_python_rpms/* That's fine as an approach for reproducing locally. Does it matter that you're installing more packages that the CI system would have ? I guess it could affect the results, but fairly slim liklihood IMHO.