[libvirt] [jenkins-ci PATCH v2 0/3] Enable building of the VZ driver on CentOS 7
The VZ driver in libvirt periodically gets broken by refactoring in libvirt. This is not noticed either before or after merge because none of our CI tests nor common developer build hosts includ the deps needed for the VZ driver. The OpenVZ project, however, does provide builds of the required packages for RHEL-7. We can use these packages in our CentOS 7 CI VMs to enable build testing of the VZ driver. This closes the only hole we have in driver build coverage for CI. Changed in v2: - Enable GPG verification - Use older repo avoid temporary deps problem - Setup repos in dockerfile too. Daniel P. Berrangé (3): guests: add openvz repository on CentOS 7 guests: define mapping for the libprlsdk package guests: add libprlsdk package to libvirt project guests/lcitool | 22 ++++++++++++++++ guests/playbooks/update/tasks/base.yml | 25 +++++++++++++++++++ guests/playbooks/update/templates/openvz.key | 20 +++++++++++++++ .../playbooks/update/templates/openvz.repo.j2 | 9 +++++++ guests/vars/mappings.yml | 4 +++ guests/vars/projects/libvirt.yml | 1 + 6 files changed, 81 insertions(+) create mode 100644 guests/playbooks/update/templates/openvz.key create mode 100644 guests/playbooks/update/templates/openvz.repo.j2 -- 2.23.0
The OpenVZ site provides a yum repo built against RHEL-7 that includes the prlsdk-devel RPM needed for the VZ driver. This repo has quite alot of packages that replace stuff from standard RHEL repos, so the yum config file is set to whitelist only the minimal RPMs we need to do builds. Fortunately they have no deps which would cause replacement of standard RHEL RPMs. Note this does not use the latest OpenVZ repo link, since that currently has broken dependencies present Error: Package: libprlcommon-7.0.183-1.vz7.x86_64 (vz) Requires: libjson-c.so.2(libjson-c.so.2)(64bit) The Requires line ought to be libjson-c.so.2()(64bit) Once that's fixed we can switch to the latest repo link. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/lcitool | 22 ++++++++++++++++ guests/playbooks/update/tasks/base.yml | 25 +++++++++++++++++++ guests/playbooks/update/templates/openvz.key | 20 +++++++++++++++ .../playbooks/update/templates/openvz.repo.j2 | 9 +++++++ 4 files changed, 76 insertions(+) create mode 100644 guests/playbooks/update/templates/openvz.key create mode 100644 guests/playbooks/update/templates/openvz.repo.j2 diff --git a/guests/lcitool b/guests/lcitool index d617beb..4f874b3 100755 --- a/guests/lcitool +++ b/guests/lcitool @@ -593,6 +593,18 @@ class Application: self._execute_playbook("build", args.hosts, args.projects, args.git_revision) + def _get_openvz_repo(self): + basedir = os.path.dirname(sys.argv[0]) + repofile = os.path.join(basedir, "playbooks", "update", "templates", "openvz.repo.j2") + with open(repofile, "r") as r: + return r.read().rstrip() + + def _get_openvz_key(self): + basedir = os.path.dirname(sys.argv[0]) + repofile = os.path.join(basedir, "playbooks", "update", "templates", "openvz.key") + with open(repofile, "r") as r: + return r.read().rstrip() + def _action_dockerfile(self, args): mappings = self._projects.get_mappings() pip_mappings = self._projects.get_pip_mappings() @@ -723,6 +735,16 @@ class Application: {package_manager} clean all -y """).format(**varmap)) elif os_name == "CentOS" and os_version == "7": + repo = self._get_openvz_repo() + repocmd = "\\n\\\n".join(repo.split("\n")) + key = self._get_openvz_key() + keycmd = "\\n\\\n".join(key.split("\n")) + + sys.stdout.write( + "RUN echo -e '%s' > /etc/yum.repos.d/openvz.repo && \\\n" % repocmd + + " echo -e '%s' > /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \\\n" % keycmd + + " rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ") + sys.stdout.write(textwrap.dedent(""" RUN {package_manager} update -y && \\ {package_manager} install -y epel-release && \\ diff --git a/guests/playbooks/update/tasks/base.yml b/guests/playbooks/update/tasks/base.yml index 3d83e78..e17b50b 100644 --- a/guests/playbooks/update/tasks/base.yml +++ b/guests/playbooks/update/tasks/base.yml @@ -13,6 +13,31 @@ package: name: epel-release state: latest + +- name: Create OpenVZ key + template: + src: '{{ playbook_base }}/templates/openvz.key' + dest: /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ + owner: root + group: root + when: + - os_name == 'CentOS' + - os_version == '7' + +- name: Import OpenVZ key + command: 'rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ' + args: + warn: no + when: + - os_name == 'CentOS' + - os_version == '7' + +- name: Enable OpenVZ repository + template: + src: '{{ playbook_base }}/templates/openvz.repo.j2' + dest: /etc/yum.repos.d/openvz.repo + owner: root + group: root when: - os_name == 'CentOS' - os_version == '7' diff --git a/guests/playbooks/update/templates/openvz.key b/guests/playbooks/update/templates/openvz.key new file mode 100644 index 0000000..b77a137 --- /dev/null +++ b/guests/playbooks/update/templates/openvz.key @@ -0,0 +1,20 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- +Version: GnuPG v2.0.22 (GNU/Linux) + +mI0EVl80nQEEAKrEeyeTCwrzS9kYedZ/sAc/GUqlb81C7pA9SaR3fyck5mVw1Ogk +YdmNBPM2kY7QDxR9F0EpSpnxSCAXZXugsQ8KzZ0DRLVeBDQyGs9IGK5hI0zzxIil +BzfvIexLiQQhLy7YlIi8Jt/uUqKkW0pIMNMGcduY97VATtczpncpkmSzABEBAAG0 +SFZpcnR1b3p6byBUZWFtIChHUEcga2V5IHNpZ25hdHVyZSBmb3IgcGFja2FnZXMp +IDxzZWN1cml0eUB2aXJ0dW96em8uY29tPoi5BBMBAgAjBQJWXzSdAhsDBwsJCAcD +AgEGFQgCCQoLBBYCAwECHgECF4AACgkQygt9GUTNrSruIgP/er70Eyo73A1gfrjv +oPUkyo4rslVRZu3qqCwoMFtJc/Z/UxWgEka1buorlcGLa6eO/EZ49c0n+KGa4Kvt +EUboIq0yEu5i0FyAj92ifm+hNhoAbGfm0cZ4/fD0oGr3l8OsQo4+iHX4xAPwFe7Y +zABuB8I1ZDZ4OIp5tDfTTuF2LT24jQRWXzSdAQQAog2Aqb+Ptl68O7cQhWLjVGkj +yyigZrdeReLx3HloKJPBeQ/kA6uvMJc/IYS3uppMWXv9v+QenS6uhP1TUJ2k9FvM +t94MQZfALN7Vpf8AF+UeWu4Ru+y4BNzcFhrPhIFNFChOR2QqW6FkgE57D9I177NC +oJMyrlNe8wcGa178An8AEQEAAYifBBgBAgAJBQJWXzSdAhsMAAoJEMoLfRlEza0q +bKwD/3+OFVIEXnIv5XgdGRNX5fHggsUN1bb8gva7HANRlKdd4LD8foDM3F/yv/3V +igG14D5EjKz56SaBDNgiI4++hOzb2M8jhAsR86jxkXFrrP1U3ZNRKg6av9DPFAPS +WEiJKtQrZDJloqtyi/mmRa1VsV7RYR0VPJjhK/R8EQ7Ysshy +=fRMg +-----END PGP PUBLIC KEY BLOCK----- diff --git a/guests/playbooks/update/templates/openvz.repo.j2 b/guests/playbooks/update/templates/openvz.repo.j2 new file mode 100644 index 0000000..19a9546 --- /dev/null +++ b/guests/playbooks/update/templates/openvz.repo.j2 @@ -0,0 +1,9 @@ +[openvz] +name=OpenVZ addons +baseurl=https://download.openvz.org/virtuozzo/releases/openvz-7.0.11-235/x86_64/os/ +enabled=1 +gpgcheck=1 +skip_if_unavailable=0 +metadata_expire=6h +priority=90 +includepkgs=libprl* -- 2.23.0
CC'ing Nikolay on this to raise the issue of broken deps in the OpenVZ repo for CentOS 7 & incorrectly documented GPG keys ... On Fri, Dec 06, 2019 at 06:53:38PM +0000, Daniel P. Berrangé wrote:
The OpenVZ site provides a yum repo built against RHEL-7 that includes the prlsdk-devel RPM needed for the VZ driver. This repo has quite alot of packages that replace stuff from standard RHEL repos, so the yum config file is set to whitelist only the minimal RPMs we need to do builds. Fortunately they have no deps which would cause replacement of standard RHEL RPMs.
Note this does not use the latest OpenVZ repo link, since that currently has broken dependencies present
Originally I was using this URL for yum: https://download.openvz.org/virtuozzo/releases/7.0/x86_64/os/ Which results in this broken dep at install time:
Error: Package: libprlcommon-7.0.183-1.vz7.x86_64 (vz) Requires: libjson-c.so.2(libjson-c.so.2)(64bit)
The Requires line ought to be
libjson-c.so.2()(64bit)
This appears to be a recent problem from the Dec 4th release of openvz-7.0.12-283 - the previous openvz-7.0.11-235 has correctly resolving deps. The other issue that I forgot to mention is that the GPG keys used for signing the RPMs on download.openvz.org are incorrectly / misleadingly documented. In the README at: https://download.openvz.org/ it documents & links to https://download.openvz.org/RPM-GPG-Key-OpenVZ saying this is used to sign RPMs on download.openvz.org This doc is repeated at https://wiki.openvz.org/Package_signatures That key has key ID a7a1d4b6 as identified as "OpenVZ Project <security@openvz.org>" This documentation is all wrong though, as this key is not used to sign the RPMs for CentOS7 at least The RPMs in https://download.openvz.org/virtuozzo/releases/7.0/x86_64/os/ at signed by key with ID 44cdad2a. It took me a long time to find this key, but eventually I discovered a link to it from https://docs.virtuozzo.com/keys/ Section 2, 2. Virtuozzo 7, Virtuozzo Automator 7, and Virtuozzo PowerPanel Signing Key https://docs.virtuozzo.com/keys/VIRTUOZZO_GPG_KEY which identifies itself as "Virtuozzo Team (GPG key signature for packages) <security@virtuozzo.com>" Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, 2019-12-06 at 18:53 +0000, Daniel P. Berrangé wrote:
+++ b/guests/lcitool + def _get_openvz_repo(self): + basedir = os.path.dirname(sys.argv[0]) + repofile = os.path.join(basedir, "playbooks", "update", "templates", "openvz.repo.j2")
This should be base = Util.get_base() repofile = os.path.join(base, ...)
+ def _get_openvz_key(self): + basedir = os.path.dirname(sys.argv[0]) + repofile = os.path.join(basedir, "playbooks", "update", "templates", "openvz.key")
Same here, except you probably want to call it keyfile instead of repofile.
@@ -723,6 +735,16 @@ class Application: elif os_name == "CentOS" and os_version == "7": + repo = self._get_openvz_repo() + repocmd = "\\n\\\n".join(repo.split("\n")) + key = self._get_openvz_key() + keycmd = "\\n\\\n".join(key.split("\n")) + + sys.stdout.write( + "RUN echo -e '%s' > /etc/yum.repos.d/openvz.repo && \\\n" % repocmd + + " echo -e '%s' > /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \\\n" % keycmd + + " rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ")
This is different from what's right above and below it for, as far as I can tell, no good reason. You can make it nicer and more consistent like repo = self._get_openvz_repo() key = self._get_openvz_key() varmap["repo"] = "\\n\\\n".join(repo.split("\n")) varmap["key"] = "\\n\\\n".join(key.split("\n")) sys.stdout.write(textwrap.dedent(""" RUN echo -e '{repo}' > /etc/yum.repos.d/openvz.repo && \\ echo -e '{key}' > /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ && \\ rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-OpenVZ """).format(**varmap)) sys.stdout.write(textwrap.dedent(""" RUN {package_manager} update -y && \\ {package_manager} install -y epel-release && \\ {package_manager} install -y {pkgs} && \\ {package_manager} autoremove -y && \\ {package_manager} clean all -y """).format(**varmap)) or even merge the two RUN statements to reduce the number of layers that will end up in the resulting image. Everything else looks good, so with the above changed Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
Add a mapping exclusively for CentOS 7 to pull in the libprlsdk package, since other distros don't have it available at this time. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/vars/mappings.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/guests/vars/mappings.yml b/guests/vars/mappings.yml index f5dab6a..ce1294c 100644 --- a/guests/vars/mappings.yml +++ b/guests/vars/mappings.yml @@ -366,6 +366,10 @@ mappings: rpm: libpciaccess-devel cross-policy-deb: foreign + libprlsdk: + default: + CentOS7: libprlsdk-devel + librbd: deb: librbd-dev Fedora: librbd-devel -- 2.23.0
On Fri, 2019-12-06 at 18:53 +0000, Daniel P. Berrangé wrote:
Add a mapping exclusively for CentOS 7 to pull in the libprlsdk package, since other distros don't have it available at this time.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/vars/mappings.yml | 4 ++++ 1 file changed, 4 insertions(+)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/vars/projects/libvirt.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/guests/vars/projects/libvirt.yml b/guests/vars/projects/libvirt.yml index 780a5aa..1efa846 100644 --- a/guests/vars/projects/libvirt.yml +++ b/guests/vars/projects/libvirt.yml @@ -31,6 +31,7 @@ packages: - libparted - libpcap - libpciaccess + - libprlsdk - librbd - libselinux - libssh -- 2.23.0
On Fri, 2019-12-06 at 18:53 +0000, Daniel P. Berrangé wrote:
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- guests/vars/projects/libvirt.yml | 1 + 1 file changed, 1 insertion(+)
Reviewed-by: Andrea Bolognani <abologna@redhat.com> -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé