[libvirt] [jenkins-ci PATCH 0/4] guests: Introduce package_manager
As mentioned in [1] and detailed in [2], Fedora is planning to drop the 'yum' command and require users to call 'dnf' exclusively. In fact, it looks like this change has already been implemented in Rawhide, so we need to adapt sooner rather than later. The first two patches address the actual compatibility issue, while the remaining ones perform some cleanups which are either made much easier by the previous changes or I just happened to stumble upon while working on this :) [1] https://www.redhat.com/archives/libvir-list/2019-May/msg00043.html [2] https://fedoraproject.org/wiki/Changes/Retire_YUM_3 Andrea Bolognani (4): guests: Introduce package_manager guests: Use package_manager everywhere guests: Don't call out to the shell twice lcitool: Fix Dockerfile alignment guests/host_vars/libvirt-centos-7/main.yml | 1 + guests/host_vars/libvirt-debian-9/main.yml | 1 + guests/host_vars/libvirt-debian-sid/main.yml | 1 + guests/host_vars/libvirt-fedora-29/main.yml | 1 + guests/host_vars/libvirt-fedora-30/main.yml | 1 + .../host_vars/libvirt-fedora-rawhide/main.yml | 1 + guests/host_vars/libvirt-freebsd-11/main.yml | 1 + guests/host_vars/libvirt-freebsd-12/main.yml | 1 + .../libvirt-freebsd-current/main.yml | 1 + guests/host_vars/libvirt-ubuntu-18/main.yml | 1 + guests/lcitool | 48 ++++++++++--------- guests/playbooks/update/tasks/base.yml | 19 ++------ guests/playbooks/update/tasks/bootstrap.yml | 16 +------ 13 files changed, 42 insertions(+), 51 deletions(-) -- 2.20.1
Platforms that share the same package format might still want to use different package managers, a good example being CentOS with yum and Fedora with dnf. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/main.yml | 1 + guests/host_vars/libvirt-debian-9/main.yml | 1 + guests/host_vars/libvirt-debian-sid/main.yml | 1 + guests/host_vars/libvirt-fedora-29/main.yml | 1 + guests/host_vars/libvirt-fedora-30/main.yml | 1 + guests/host_vars/libvirt-fedora-rawhide/main.yml | 1 + guests/host_vars/libvirt-freebsd-11/main.yml | 1 + guests/host_vars/libvirt-freebsd-12/main.yml | 1 + guests/host_vars/libvirt-freebsd-current/main.yml | 1 + guests/host_vars/libvirt-ubuntu-18/main.yml | 1 + 10 files changed, 10 insertions(+) diff --git a/guests/host_vars/libvirt-centos-7/main.yml b/guests/host_vars/libvirt-centos-7/main.yml index fa4fc67..94e29af 100644 --- a/guests/host_vars/libvirt-centos-7/main.yml +++ b/guests/host_vars/libvirt-centos-7/main.yml @@ -13,6 +13,7 @@ projects: - virt-viewer package_format: 'rpm' +package_manager: 'yum' os_name: 'CentOS' os_version: '7' diff --git a/guests/host_vars/libvirt-debian-9/main.yml b/guests/host_vars/libvirt-debian-9/main.yml index ec7e6b4..6b685a4 100644 --- a/guests/host_vars/libvirt-debian-9/main.yml +++ b/guests/host_vars/libvirt-debian-9/main.yml @@ -17,6 +17,7 @@ projects: - virt-viewer package_format: 'deb' +package_manager: 'apt-get' os_name: 'Debian' os_version: '9' diff --git a/guests/host_vars/libvirt-debian-sid/main.yml b/guests/host_vars/libvirt-debian-sid/main.yml index 1c7a29b..3808383 100644 --- a/guests/host_vars/libvirt-debian-sid/main.yml +++ b/guests/host_vars/libvirt-debian-sid/main.yml @@ -17,6 +17,7 @@ projects: - virt-viewer package_format: 'deb' +package_manager: 'apt-get' os_name: 'Debian' os_version: 'Sid' diff --git a/guests/host_vars/libvirt-fedora-29/main.yml b/guests/host_vars/libvirt-fedora-29/main.yml index bebf171..ac0228d 100644 --- a/guests/host_vars/libvirt-fedora-29/main.yml +++ b/guests/host_vars/libvirt-fedora-29/main.yml @@ -18,6 +18,7 @@ projects: - virt-viewer package_format: 'rpm' +package_manager: 'dnf' os_name: 'Fedora' os_version: '29' diff --git a/guests/host_vars/libvirt-fedora-30/main.yml b/guests/host_vars/libvirt-fedora-30/main.yml index 4ad27a6..491b112 100644 --- a/guests/host_vars/libvirt-fedora-30/main.yml +++ b/guests/host_vars/libvirt-fedora-30/main.yml @@ -18,6 +18,7 @@ projects: - virt-viewer package_format: 'rpm' +package_manager: 'dnf' os_name: 'Fedora' os_version: '30' diff --git a/guests/host_vars/libvirt-fedora-rawhide/main.yml b/guests/host_vars/libvirt-fedora-rawhide/main.yml index ed0a3fa..db46825 100644 --- a/guests/host_vars/libvirt-fedora-rawhide/main.yml +++ b/guests/host_vars/libvirt-fedora-rawhide/main.yml @@ -28,6 +28,7 @@ projects: - virt-viewer+mingw64 package_format: 'rpm' +package_manager: 'dnf' os_name: 'Fedora' os_version: 'Rawhide' diff --git a/guests/host_vars/libvirt-freebsd-11/main.yml b/guests/host_vars/libvirt-freebsd-11/main.yml index ed805c9..e9f6d03 100644 --- a/guests/host_vars/libvirt-freebsd-11/main.yml +++ b/guests/host_vars/libvirt-freebsd-11/main.yml @@ -16,6 +16,7 @@ projects: - virt-viewer package_format: 'pkg' +package_manager: 'pkg' os_name: 'FreeBSD' os_version: '11' diff --git a/guests/host_vars/libvirt-freebsd-12/main.yml b/guests/host_vars/libvirt-freebsd-12/main.yml index 8bbe158..ba3ba62 100644 --- a/guests/host_vars/libvirt-freebsd-12/main.yml +++ b/guests/host_vars/libvirt-freebsd-12/main.yml @@ -16,6 +16,7 @@ projects: - virt-viewer package_format: 'pkg' +package_manager: 'pkg' os_name: 'FreeBSD' os_version: '12' diff --git a/guests/host_vars/libvirt-freebsd-current/main.yml b/guests/host_vars/libvirt-freebsd-current/main.yml index 62498fd..74e1856 100644 --- a/guests/host_vars/libvirt-freebsd-current/main.yml +++ b/guests/host_vars/libvirt-freebsd-current/main.yml @@ -16,6 +16,7 @@ projects: - virt-viewer package_format: 'pkg' +package_manager: 'pkg' os_name: 'FreeBSD' os_version: 'Current' diff --git a/guests/host_vars/libvirt-ubuntu-18/main.yml b/guests/host_vars/libvirt-ubuntu-18/main.yml index bb465e8..5b5bf0c 100644 --- a/guests/host_vars/libvirt-ubuntu-18/main.yml +++ b/guests/host_vars/libvirt-ubuntu-18/main.yml @@ -17,6 +17,7 @@ projects: - virt-viewer package_format: 'deb' +package_manager: 'apt-get' os_name: 'Ubuntu' os_version: '18' -- 2.20.1
On Tue, May 07, 2019 at 03:17:39PM +0200, Andrea Bolognani wrote:
Platforms that share the same package format might still want to use different package managers, a good example being CentOS with yum and Fedora with dnf.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/host_vars/libvirt-centos-7/main.yml | 1 + guests/host_vars/libvirt-debian-9/main.yml | 1 + guests/host_vars/libvirt-debian-sid/main.yml | 1 + guests/host_vars/libvirt-fedora-29/main.yml | 1 + guests/host_vars/libvirt-fedora-30/main.yml | 1 + guests/host_vars/libvirt-fedora-rawhide/main.yml | 1 + guests/host_vars/libvirt-freebsd-11/main.yml | 1 + guests/host_vars/libvirt-freebsd-12/main.yml | 1 + guests/host_vars/libvirt-freebsd-current/main.yml | 1 + guests/host_vars/libvirt-ubuntu-18/main.yml | 1 + 10 files changed, 10 insertions(+)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Instead of hardcoding the name of the package manager in commands, use the value obtained from the inventory. In some cases this is necessary, eg. when RPM-based distributions are involved; for most other cases we could get away with keepking the hardcoded names, but it's better to be completely consistent to hopefully avoid usage of the wrong package manager slipping with further changes. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 40 +++++++++++---------- guests/playbooks/update/tasks/base.yml | 12 +++---- guests/playbooks/update/tasks/bootstrap.yml | 16 ++------- 3 files changed, 29 insertions(+), 39 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index 60a01fc..c179775 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -591,6 +591,7 @@ class Application: facts = self._inventory.get_facts(host) package_format = facts["package_format"] + package_manager = facts["package_manager"] os_name = facts["os_name"] os_version = facts["os_version"] os_full = os_name + os_version @@ -651,6 +652,7 @@ class Application: print("FROM {}".format(facts["docker_base"])) varmap = {} + varmap["package_manager"] = package_manager varmap["pkgs"] = " \\\n ".join(sorted(set(pkgs.values()))) if package_format == "deb": if args.cross_arch: @@ -667,12 +669,12 @@ class Application: sys.stdout.write(textwrap.dedent(""" RUN export DEBIAN_FRONTEND=noninteractive && \\ - apt-get update && \\ - apt-get dist-upgrade -y && \\ - apt-get install --no-install-recommends -y \\ + {package_manager} update && \\ + {package_manager} dist-upgrade -y && \\ + {package_manager} install --no-install-recommends -y \\ {pkgs} && \\ - apt-get autoremove -y && \\ - apt-get autoclean -y + {package_manager} autoremove -y && \\ + {package_manager} autoclean -y """).format(**varmap)) if args.cross_arch: # Intentionally a separate RUN command from the above @@ -681,12 +683,12 @@ class Application: sys.stdout.write(textwrap.dedent(""" RUN export DEBIAN_FRONTEND=noninteractive && \\ dpkg --add-architecture {cross_arch} && \\ - apt-get update && \\ - apt-get dist-upgrade -y && \\ - apt-get install --no-install-recommends -y \\ + {package_manager} update && \\ + {package_manager} dist-upgrade -y && \\ + {package_manager} install --no-install-recommends -y \\ {cross_pkgs} && \\ - apt-get autoremove -y && \\ - apt-get autoclean -y + {package_manager} autoremove -y && \\ + {package_manager} autoclean -y ENV ABI "{cross_abi}" ENV CONFIGURE_OPTS "--host={cross_abi} \\ @@ -696,18 +698,18 @@ class Application: elif package_format == "rpm": if os_name == "Fedora" and os_version == "Rawhide": sys.stdout.write(textwrap.dedent(""" - RUN yum update -y --nogpgcheck fedora-gpg-keys && \\ - yum update -y && \\ - yum install -y {pkgs} && \\ - yum autoremove -y && \\ - yum clean all -y + RUN {package_manager} update -y --nogpgcheck fedora-gpg-keys && \\ + {package_manager} update -y && \\ + {package_manager} install -y {pkgs} && \\ + {package_manager} autoremove -y && \\ + {package_manager} clean all -y """).format(**varmap)) else: sys.stdout.write(textwrap.dedent(""" - RUN yum update -y && \\ - yum install -y {pkgs} && \\ - yum autoremove -y && \\ - yum clean all -y + RUN {package_manager} update -y && \\ + {package_manager} install -y {pkgs} && \\ + {package_manager} autoremove -y && \\ + {package_manager} clean all -y """).format(**varmap)) def run(self): diff --git a/guests/playbooks/update/tasks/base.yml b/guests/playbooks/update/tasks/base.yml index d71d99f..59f6124 100644 --- a/guests/playbooks/update/tasks/base.yml +++ b/guests/playbooks/update/tasks/base.yml @@ -28,7 +28,7 @@ - os_version == 'Rawhide' - name: Update installed packages - command: dnf update --refresh --exclude 'kernel*' -y + command: '{{ package_manager }} update --refresh --exclude "kernel*" -y' args: warn: no when: @@ -36,7 +36,7 @@ - os_version == 'Rawhide' - name: Update installed packages - command: dnf update --disablerepo='*' --enablerepo=fedora-rawhide-kernel-nodebug 'kernel*' -y + command: '{{ package_manager }} update --disablerepo="*" --enablerepo=fedora-rawhide-kernel-nodebug "kernel*" -y' args: warn: no when: @@ -51,19 +51,19 @@ - package_format == 'deb' - name: Update installed packages - shell: pkg update && pkg upgrade -y + shell: '{{ package_manager }} update && {{ package_manager }} upgrade -y' when: - package_format == 'pkg' - name: Clean up packages after update - command: yum clean packages -y + command: '{{ package_manager }} clean packages -y' args: warn: no when: - package_format == 'rpm' - name: Clean up packages after update - command: yum autoremove -y + command: '{{ package_manager }} autoremove -y' args: warn: no when: @@ -77,7 +77,7 @@ - package_format == 'deb' - name: Clean up packages after update - shell: pkg clean -y && pkg autoremove -y + shell: '{{ package_manager }} clean -y && {{ package_manager }} autoremove -y' when: - package_format == 'pkg' diff --git a/guests/playbooks/update/tasks/bootstrap.yml b/guests/playbooks/update/tasks/bootstrap.yml index 44bb2c9..2e1dc42 100644 --- a/guests/playbooks/update/tasks/bootstrap.yml +++ b/guests/playbooks/update/tasks/bootstrap.yml @@ -1,6 +1,6 @@ --- - name: Bootstrap the pkgng package manager - raw: env ASSUME_ALWAYS_YES=YES pkg bootstrap + raw: 'env ASSUME_ALWAYS_YES=YES {{ package_manager }} bootstrap' when: - package_format == 'pkg' @@ -14,16 +14,4 @@ - os_version == '7' - name: Bootstrap Ansible - raw: 'yum install -y {{ python }}' - when: - - package_format == 'rpm' - -- name: Bootstrap Ansible - raw: 'apt-get install -y {{ python }}' - when: - - package_format == 'deb' - -- name: Bootstrap Ansible - raw: 'pkg install -y {{ python }}' - when: - - package_format == 'pkg' + raw: '{{ package_manager }} install -y {{ python }}' -- 2.20.1
On Tue, May 07, 2019 at 03:17:40PM +0200, Andrea Bolognani wrote:
Instead of hardcoding the name of the package manager in commands, use the value obtained from the inventory.
In some cases this is necessary, eg. when RPM-based distributions are involved; for most other cases we could get away with keepking the hardcoded names, but it's better to be completely consistent to hopefully avoid usage of the wrong package manager slipping with further changes.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 40 +++++++++++---------- guests/playbooks/update/tasks/base.yml | 12 +++---- guests/playbooks/update/tasks/bootstrap.yml | 16 ++------- 3 files changed, 29 insertions(+), 39 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
- name: Bootstrap Ansible - raw: 'yum install -y {{ python }}' - when: - - package_format == 'rpm' - -- name: Bootstrap Ansible - raw: 'apt-get install -y {{ python }}' - when: - - package_format == 'deb' - -- name: Bootstrap Ansible - raw: 'pkg install -y {{ python }}' - when: - - package_format == 'pkg' + raw: '{{ package_manager }} install -y {{ python }}'
Heh, rather amuzing that all three package managers happen to support the same "install -y" command syntax. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, 2019-05-09 at 13:08 +0100, Daniel P. Berrangé wrote:
On Tue, May 07, 2019 at 03:17:40PM +0200, Andrea Bolognani wrote:
- name: Bootstrap Ansible - raw: 'yum install -y {{ python }}' - when: - - package_format == 'rpm' - -- name: Bootstrap Ansible - raw: 'apt-get install -y {{ python }}' - when: - - package_format == 'deb' - -- name: Bootstrap Ansible - raw: 'pkg install -y {{ python }}' - when: - - package_format == 'pkg' + raw: '{{ package_manager }} install -y {{ python }}'
Heh, rather amuzing that all three package managers happen to support the same "install -y" command syntax.
"We don't make mistakes, just happy little accidents." :) -- Andrea Bolognani / Red Hat / Virtualization
We're already doing this for FreeBSD, so do it for CentOS and Fedora as well. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/playbooks/update/tasks/base.yml | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/guests/playbooks/update/tasks/base.yml b/guests/playbooks/update/tasks/base.yml index 59f6124..8fe114e 100644 --- a/guests/playbooks/update/tasks/base.yml +++ b/guests/playbooks/update/tasks/base.yml @@ -56,16 +56,7 @@ - package_format == 'pkg' - name: Clean up packages after update - command: '{{ package_manager }} clean packages -y' - args: - warn: no - when: - - package_format == 'rpm' - -- name: Clean up packages after update - command: '{{ package_manager }} autoremove -y' - args: - warn: no + shell: '{{ package_manager }} clean packages -y && {{ package_manager }} autoremove -y' when: - package_format == 'rpm' -- 2.20.1
On Tue, May 07, 2019 at 03:17:41PM +0200, Andrea Bolognani wrote:
We're already doing this for FreeBSD, so do it for CentOS and Fedora as well.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/playbooks/update/tasks/base.yml | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Now that we are using package_manager everywhere instead of hardcoding the names, it's finally possible to make the alignment of the resulting Dockerfiles perfect. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/guests/lcitool b/guests/lcitool index c179775..5cf8efe 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -649,11 +649,13 @@ class Application: if pkgs[package] is None or cross_policy in ["skip", "foreign"]: del pkgs[package] + pkg_align = " \\\n" + (" " * len("RUN " + package_manager + " ")) + print("FROM {}".format(facts["docker_base"])) varmap = {} varmap["package_manager"] = package_manager - varmap["pkgs"] = " \\\n ".join(sorted(set(pkgs.values()))) + varmap["pkgs"] = pkg_align[1:] + pkg_align.join(sorted(set(pkgs.values()))) if package_format == "deb": if args.cross_arch: deb_arch = Util.native_arch_to_deb_arch(args.cross_arch) @@ -663,7 +665,7 @@ class Application: varmap["cross_arch"] = deb_arch pkg_names = [p + ":" + deb_arch for p in cross_pkgs.values()] pkg_names.append(gcc) - varmap["cross_pkgs"] = " \\\n ".join(sorted(set(pkg_names))) + varmap["cross_pkgs"] = pkg_align[1:] + pkg_align.join(sorted(set(pkg_names))) varmap["cross_abi"] = abi varmap["cross_lib"] = lib @@ -671,8 +673,7 @@ class Application: RUN export DEBIAN_FRONTEND=noninteractive && \\ {package_manager} update && \\ {package_manager} dist-upgrade -y && \\ - {package_manager} install --no-install-recommends -y \\ - {pkgs} && \\ + {package_manager} install --no-install-recommends -y {pkgs} && \\ {package_manager} autoremove -y && \\ {package_manager} autoclean -y """).format(**varmap)) @@ -685,8 +686,7 @@ class Application: dpkg --add-architecture {cross_arch} && \\ {package_manager} update && \\ {package_manager} dist-upgrade -y && \\ - {package_manager} install --no-install-recommends -y \\ - {cross_pkgs} && \\ + {package_manager} install --no-install-recommends -y {cross_pkgs} && \\ {package_manager} autoremove -y && \\ {package_manager} autoclean -y -- 2.20.1
On Tue, May 07, 2019 at 03:17:42PM +0200, Andrea Bolognani wrote:
Now that we are using package_manager everywhere instead of hardcoding the names, it's finally possible to make the alignment of the resulting Dockerfiles perfect.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- guests/lcitool | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé