[libvirt] [jenkins-ci PATCH 0/5] lcitool: Make Dockerfile generation work on openSUSE
With this series applied, running ./lcitool dockerfile libvirt-opensuse-151 libvirt results in a Dockerfile that can be successfully used to build a container image, and that image in turn can be successfully used to build libvirt in. Andrea Bolognani (5): guests: Add mapping for augeas-lenses guests: Install augeas-lenses for libvirt lcitool: Remove redundant update step for openSUSE guests: Add base container image for openSUSE Leap 15.1 lcitool: Make Dockerfile generation work on openSUSE guests/host_vars/libvirt-opensuse-151/docker.yml | 2 ++ guests/lcitool | 14 ++++++++++++-- guests/playbooks/update/tasks/base.yml | 7 ------- guests/vars/mappings.yml | 5 +++++ guests/vars/projects/libvirt.yml | 1 + 5 files changed, 20 insertions(+), 9 deletions(-) create mode 100644 guests/host_vars/libvirt-opensuse-151/docker.yml -- 2.23.0
Some distributions package augeas and the default lenses separately, so we need a mapping for the latter. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/vars/mappings.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index d32ecf3..02bedab 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -77,6 +77,11 @@ mappings: default: augeas deb: augeas-tools + augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses + autoconf: default: autoconf -- 2.23.0
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
Some distributions package augeas and the default lenses separately, so we need a mapping for the latter.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/vars/mappings.yml | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index d32ecf3..02bedab 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -77,6 +77,11 @@ mappings: default: augeas deb: augeas-tools
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Hmm. I guess it only hits the container, am I right? Because in the VM it's installed as part of the base system.
On Thu, 2019-12-12 at 15:43 +0100, Fabiano Fidêncio wrote:
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Hmm. I guess it only hits the container, am I right? Because in the VM it's installed as part of the base system.
It looks like it's needed by yast: $ sudo zypper remove augeas-lenses Loading repository data... Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed. Reading installed packages... Resolving package dependencies... The following 15 packages are going to be REMOVED: augeas-lenses patterns-yast-yast2_basis yast2-bootloader yast2-country yast2-installation yast2-ldap yast2-mail yast2-network yast2-online-update yast2-online-update-frontend yast2-packager yast2-storage-ng yast2-tune yast2-update yast2-users The following pattern is going to be REMOVED: yast2_basis 15 packages to remove. After the operation, 7.5 MiB will be freed. Continue? [y/n/v/...? shows all options] (y): The container is minimal, and doesn't include yast at all: $ rpm -qa | grep yast $ So that would explain the difference. -- Andrea Bolognani / Red Hat / Virtualization
On 12/12/19 8:07 AM, Andrea Bolognani wrote:
On Thu, 2019-12-12 at 15:43 +0100, Fabiano Fidêncio wrote:
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Hmm. I guess it only hits the container, am I right? Because in the VM it's installed as part of the base system.
It looks like it's needed by yast:
$ sudo zypper remove augeas-lenses Loading repository data... Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed. Reading installed packages... Resolving package dependencies...
The following 15 packages are going to be REMOVED: augeas-lenses patterns-yast-yast2_basis yast2-bootloader yast2-country yast2-installation yast2-ldap yast2-mail yast2-network yast2-online-update yast2-online-update-frontend yast2-packager yast2-storage-ng yast2-tune yast2-update yast2-users
The following pattern is going to be REMOVED: yast2_basis
15 packages to remove. After the operation, 7.5 MiB will be freed. Continue? [y/n/v/...? shows all options] (y):
The container is minimal, and doesn't include yast at all:
$ rpm -qa | grep yast $
So that would explain the difference.
If the difference is unwanted we could remove the yast2_basis pattern from autoinst.xml. Regards, Jim
On Thu, 2019-12-12 at 15:57 +0000, Jim Fehlig wrote:
On 12/12/19 8:07 AM, Andrea Bolognani wrote:
On Thu, 2019-12-12 at 15:43 +0100, Fabiano Fidêncio wrote:
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Hmm. I guess it only hits the container, am I right? Because in the VM it's installed as part of the base system.
It looks like it's needed by yast:
$ sudo zypper remove augeas-lenses Loading repository data... Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed. Reading installed packages... Resolving package dependencies...
The following 15 packages are going to be REMOVED: augeas-lenses patterns-yast-yast2_basis yast2-bootloader yast2-country yast2-installation yast2-ldap yast2-mail yast2-network yast2-online-update yast2-online-update-frontend yast2-packager yast2-storage-ng yast2-tune yast2-update yast2-users
The following pattern is going to be REMOVED: yast2_basis
15 packages to remove. After the operation, 7.5 MiB will be freed. Continue? [y/n/v/...? shows all options] (y):
The container is minimal, and doesn't include yast at all:
$ rpm -qa | grep yast $
So that would explain the difference.
If the difference is unwanted we could remove the yast2_basis pattern from autoinst.xml.
What would we lose by doing so? In general we install only the very basics through the OS' native installer and then add everything we need on top using lcitool, so it certainly sounds like it could make sense to drop it... -- Andrea Bolognani / Red Hat / Virtualization
On 12/12/19 9:19 AM, Andrea Bolognani wrote:
On Thu, 2019-12-12 at 15:57 +0000, Jim Fehlig wrote:
On 12/12/19 8:07 AM, Andrea Bolognani wrote:
On Thu, 2019-12-12 at 15:43 +0100, Fabiano Fidêncio wrote:
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Hmm. I guess it only hits the container, am I right? Because in the VM it's installed as part of the base system.
It looks like it's needed by yast:
$ sudo zypper remove augeas-lenses Loading repository data... Warning: No repositories defined. Operating only with the installed resolvables. Nothing can be installed. Reading installed packages... Resolving package dependencies...
The following 15 packages are going to be REMOVED: augeas-lenses patterns-yast-yast2_basis yast2-bootloader yast2-country yast2-installation yast2-ldap yast2-mail yast2-network yast2-online-update yast2-online-update-frontend yast2-packager yast2-storage-ng yast2-tune yast2-update yast2-users
The following pattern is going to be REMOVED: yast2_basis
15 packages to remove. After the operation, 7.5 MiB will be freed. Continue? [y/n/v/...? shows all options] (y):
The container is minimal, and doesn't include yast at all:
$ rpm -qa | grep yast $
So that would explain the difference.
If the difference is unwanted we could remove the yast2_basis pattern from autoinst.xml.
What would we lose by doing so? In general we install only the very basics through the OS' native installer and then add everything we need on top using lcitool, so it certainly sounds like it could make sense to drop it...
It's an installation and config management tool. Obviously we don't need it for installation, and I don't think we'll be using it for management within the VM. I can try removing it and send a patch if there are no ill effects, but it will have to wait until next week when I have some free time. Regards, Jim
On Thu, 2019-12-12 at 16:49 +0000, Jim Fehlig wrote:
On 12/12/19 9:19 AM, Andrea Bolognani wrote:
On Thu, 2019-12-12 at 15:57 +0000, Jim Fehlig wrote:
If the difference is unwanted we could remove the yast2_basis pattern from autoinst.xml.
What would we lose by doing so? In general we install only the very basics through the OS' native installer and then add everything we need on top using lcitool, so it certainly sounds like it could make sense to drop it...
It's an installation and config management tool. Obviously we don't need it for installation, and I don't think we'll be using it for management within the VM.
Yeah, dropping it sounds like a good idea then.
I can try removing it and send a patch if there are no ill effects, but it will have to wait until next week when I have some free time.
Take your time... Someone else might even beat you to it ;) -- Andrea Bolognani / Red Hat / Virtualization
On Thursday, 12 December 2019 15:31:04 CET Andrea Bolognani wrote:
Some distributions package augeas and the default lenses separately, so we need a mapping for the latter.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/vars/mappings.yml | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index d32ecf3..02bedab 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -77,6 +77,11 @@ mappings: default: augeas deb: augeas-tools
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Why is this needed? The lenses must be already a dependency of the augeas shared library, or at least of the tools. -- Pino Toscano
On Thu, 2019-12-12 at 16:05 +0100, Pino Toscano wrote:
On Thursday, 12 December 2019 15:31:04 CET Andrea Bolognani wrote:
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Why is this needed? The lenses must be already a dependency of the augeas shared library, or at least of the tools.
Not on openSUSE, apparently: $ docker run --rm -it opensuse/leap:15.1 edb149088975:/ # zypper install augeas Building repository 'Non-OSS Repository' cache ...................... Building repository 'Main Repository' cache ......................... Building repository 'Main Update Repository' cache .................. Building repository 'Update Repository (Non-Oss)' cache ............. Loading repository data... Reading installed packages... Resolving package dependencies... The following NEW package is going to be installed: augeas 1 new package to install. Overall download size: 130.1 KiB. Already cached: 0 B. After the operation, additional 205.1 KiB will be used. Continue? [y/n/v/...? shows all options] (y): y Retrieving package augeas-1.10.1-lp151.2.3.x86_64 (1/1), 130.1 KiB (205.1 KiB unpacked) Retrieving: augeas-1.10.1-lp151.2.3.x86_64.rpm ...................... Checking for file conflicts: ........................................ (1/1) Installing: augeas-1.10.1-lp151.2.3.x86_64 .................... edb149088975:/ # rpm -qa | grep augeas-lenses edb149088975:/ # ls /usr/share/augeas/lenses ls: cannot access '/usr/share/augeas/lenses': No such file or directory edb149088975:/ # ¯\_(ツ)_/¯ -- Andrea Bolognani / Red Hat / Virtualization
On Thursday, 12 December 2019 16:26:26 CET Andrea Bolognani wrote:
On Thu, 2019-12-12 at 16:05 +0100, Pino Toscano wrote:
On Thursday, 12 December 2019 15:31:04 CET Andrea Bolognani wrote:
+ augeas-lenses: + default: augeas + deb: augeas-lenses + OpenSUSE: augeas-lenses
Why is this needed? The lenses must be already a dependency of the augeas shared library, or at least of the tools.
Not on openSUSE, apparently:
$ docker run --rm -it opensuse/leap:15.1 edb149088975:/ # zypper install augeas Building repository 'Non-OSS Repository' cache ...................... Building repository 'Main Repository' cache ......................... Building repository 'Main Update Repository' cache .................. Building repository 'Update Repository (Non-Oss)' cache ............. Loading repository data... Reading installed packages... Resolving package dependencies...
The following NEW package is going to be installed: augeas
1 new package to install. Overall download size: 130.1 KiB. Already cached: 0 B. After the operation, additional 205.1 KiB will be used. Continue? [y/n/v/...? shows all options] (y): y Retrieving package augeas-1.10.1-lp151.2.3.x86_64 (1/1), 130.1 KiB (205.1 KiB unpacked) Retrieving: augeas-1.10.1-lp151.2.3.x86_64.rpm ......................
Checking for file conflicts: ........................................ (1/1) Installing: augeas-1.10.1-lp151.2.3.x86_64 .................... edb149088975:/ # rpm -qa | grep augeas-lenses edb149088975:/ # ls /usr/share/augeas/lenses ls: cannot access '/usr/share/augeas/lenses': No such file or directory edb149088975:/ #
Sigh... OTOH, in the mapping file I see: augeas: default: augeas deb: augeas-tools This means that most probably there are different packages installed depending on the distro: - on Debian-based distros you get the command line tools - on some distros (mostly RPM-based) you get only the shared library - on some other distros (FreeBSD, and Linux distros not handled yet) you may get either everything, or the command line tools OTOH², looking at the spec [1] makes me sad, as nothing really depends on augeas-lenses, neither the shared library package nor the tools package :-( This is totally nonsense, you install augeas in openSUSE and it is broken by default if you have recommends disabled... So my suggestion is to limit the manual lenses installation only on openSUSE, as any other distro is already doing a sane job in this case. [1] https://build.opensuse.org/package/view_file/openSUSE:Factory/augeas/augeas.... -- Pino Toscano
On Thu, 2019-12-12 at 16:42 +0100, Pino Toscano wrote:
OTOH, in the mapping file I see:
augeas: default: augeas deb: augeas-tools
This means that most probably there are different packages installed depending on the distro: - on Debian-based distros you get the command line tools - on some distros (mostly RPM-based) you get only the shared library - on some other distros (FreeBSD, and Linux distros not handled yet) you may get either everything, or the command line tools
I guess a better name for the mapping would indeed be 'augtool', because that's what we're trying to install. And it works as intended, even on CentOS and Fedora despite what you wrote above.
OTOH², looking at the spec [1] makes me sad, as nothing really depends on augeas-lenses, neither the shared library package nor the tools package :-( This is totally nonsense, you install augeas in openSUSE and it is broken by default if you have recommends disabled...
YaST2 apparently requires augeas-lenses[1], so while I agree that the current situation is probably broken the issue will not actually not show up in most openSUSE installations.
So my suggestion is to limit the manual lenses installation only on openSUSE, as any other distro is already doing a sane job in this case.
We usually don't do that. Even though we would expect that installing augtool would bring in the lenses as well, there's really nothing inherently wrong with installing the lenses explicitly: different distros have different granularity, so it's quite common that we end up resolving multiple mappings to the same concrete package, and in fact we have code in place to deal with just that situation. tl;dr openSUSE might want to fix their dependencies, but as far as I'm concerned the current situation in libvirt-jenkins-ci is perfectly fine and I'm in no hurry to change it. [1] https://www.redhat.com/archives/libvir-list/2019-December/msg00858.html -- Andrea Bolognani / Red Hat / Virtualization
libvirt's 'make check' requires the default lenses to be available. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/vars/projects/libvirt.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/guests/vars/projects/libvirt.yml b/guests/vars/projects/libvirt.yml index 780a5aa..b35e44a 100644 --- a/guests/vars/projects/libvirt.yml +++ b/guests/vars/projects/libvirt.yml @@ -2,6 +2,7 @@ packages: - apparmor - augeas + - augeas-lenses - avahi - cyrus-sasl - device-mapper -- 2.23.0
We already use the package module to update all RPM-based distros just a few lines above. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/playbooks/update/tasks/base.yml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/guests/playbooks/update/tasks/base.yml b/guests/playbooks/update/tasks/base.yml index b3ab738..188ef0a 100644 --- a/guests/playbooks/update/tasks/base.yml +++ b/guests/playbooks/update/tasks/base.yml @@ -72,13 +72,6 @@ when: - package_format == 'pkg' -- name: Update installed packages - command: '{{ package_manager }} update -y -l --force-resolution --no-recommends' - args: - warn: no - when: - - os_name == 'OpenSUSE' - - name: Clean up packages after update shell: '{{ package_manager }} clean packages -y && {{ package_manager }} autoremove -y' args: -- 2.23.0
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-opensuse-151/docker.yml | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 guests/host_vars/libvirt-opensuse-151/docker.yml diff --git a/guests/host_vars/libvirt-opensuse-151/docker.yml b/guests/host_vars/libvirt-opensuse-151/docker.yml new file mode 100644 index 0000000..6ba7adb --- /dev/null +++ b/guests/host_vars/libvirt-opensuse-151/docker.yml @@ -0,0 +1,2 @@ +--- +docker_base: opensuse/leap:15.1 -- 2.23.0
We can't use the same command sequence as for CentOS and Fedora. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 9958508..d862282 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -767,10 +767,20 @@ class Application: commands.extend([ "{package_manager} update -y", "{package_manager} install -y {pkgs}", - "{package_manager} autoremove -y", - "{package_manager} clean all -y", ]) + # openSUSE doesn't seem to have a convenient way to remove all + # unnecessary packages, but CentOS and Fedora do + if os_name == "OpenSUSE": + commands.extend([ + "{package_manager} clean --all", + ]) + else: + commands.extend([ + "{package_manager} autoremove -y", + "{package_manager} clean all -y", + ]) + script = "\nRUN " + (" && \\\n ".join(commands)) + "\n" sys.stdout.write(script.format(**varmap)) -- 2.23.0
[snip]
+ # openSUSE doesn't seem to have a convenient way to remove all + # unnecessary packages, but CentOS and Fedora do + if os_name == "OpenSUSE": + commands.extend([ + "{package_manager} clean --all", + ]) + else: + commands.extend([ + "{package_manager} autoremove -y", + "{package_manager} clean all -y", + ]) +
IMHO, it'd be easier to follow / cleaner if we do: if os_name != "OpenSUSE:" commands.extend([ "{package_manager} autoremove -y", ]) And then, later on ... commands.extend([ "{package_manager} clean all -y", ]) [snip] Best Regards, -- Fabiano Fidêncio
On Thu, 2019-12-12 at 15:39 +0100, Fabiano Fidêncio wrote:
+ # openSUSE doesn't seem to have a convenient way to remove all + # unnecessary packages, but CentOS and Fedora do + if os_name == "OpenSUSE": + commands.extend([ + "{package_manager} clean --all", + ]) + else: + commands.extend([ + "{package_manager} autoremove -y", + "{package_manager} clean all -y", + ]) +
IMHO, it'd be easier to follow / cleaner if we do: if os_name != "OpenSUSE:" commands.extend([ "{package_manager} autoremove -y", ])
And then, later on ... commands.extend([ "{package_manager} clean all -y", ])
Nope, that wouldn't work: the subcommand is clean --all on openSUSE and clean all -y everywhere else, soo there's no part that's common. -- Andrea Bolognani / Red Hat / Virtualization
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
With this series applied, running
./lcitool dockerfile libvirt-opensuse-151 libvirt
results in a Dockerfile that can be successfully used to build a container image, and that image in turn can be successfully used to build libvirt in.
Andrea Bolognani (5): guests: Add mapping for augeas-lenses guests: Install augeas-lenses for libvirt lcitool: Remove redundant update step for openSUSE guests: Add base container image for openSUSE Leap 15.1 lcitool: Make Dockerfile generation work on openSUSE
guests/host_vars/libvirt-opensuse-151/docker.yml | 2 ++ guests/lcitool | 14 ++++++++++++-- guests/playbooks/update/tasks/base.yml | 7 ------- guests/vars/mappings.yml | 5 +++++ guests/vars/projects/libvirt.yml | 1 + 5 files changed, 20 insertions(+), 9 deletions(-) create mode 100644 guests/host_vars/libvirt-opensuse-151/docker.yml
-- 2.23.0
There's one suggestion in the last patch. Please, take a look at that and feel free to adopt the suggestion or not. Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>
On Thu, 2019-12-12 at 15:45 +0100, Fabiano Fidêncio wrote:
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
Andrea Bolognani (5): guests: Add mapping for augeas-lenses guests: Install augeas-lenses for libvirt lcitool: Remove redundant update step for openSUSE guests: Add base container image for openSUSE Leap 15.1 lcitool: Make Dockerfile generation work on openSUSE
There's one suggestion in the last patch. Please, take a look at that and feel free to adopt the suggestion or not.
Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>
Thanks, I've pushed the series now. I'm confident Jim will point out any silly mistake I might have made because of my lack of experience with openSUSE :) -- Andrea Bolognani / Red Hat / Virtualization
On 12/12/19 8:31 AM, Andrea Bolognani wrote:
On Thu, 2019-12-12 at 15:45 +0100, Fabiano Fidêncio wrote:
On Thu, Dec 12, 2019 at 3:31 PM Andrea Bolognani <abologna@redhat.com> wrote:
Andrea Bolognani (5): guests: Add mapping for augeas-lenses guests: Install augeas-lenses for libvirt lcitool: Remove redundant update step for openSUSE guests: Add base container image for openSUSE Leap 15.1 lcitool: Make Dockerfile generation work on openSUSE
There's one suggestion in the last patch. Please, take a look at that and feel free to adopt the suggestion or not.
Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>
Thanks, I've pushed the series now. I'm confident Jim will point out any silly mistake I might have made because of my lack of experience with openSUSE :)
It looks good to me! Thanks for taking care of one of the tasks on my list :-). Regards, Jim
participants (4)
-
Andrea Bolognani -
Fabiano Fidêncio -
Jim Fehlig -
Pino Toscano