[libvirt PATCH 0/5] ci: Add an integration test job utilizing upstream QEMU
Since QEMU doesn't maintain a spec file in upstream, we cannot build RPM artifacts as part of the CI as we do for libvirt. Instead of hard-coding the build steps for QEMU though patch 3/5 pulls in QEMU's CI job template which means we'll remain in sync if QEMU makes changes to its build process. Erik Skultety (5): ci: Separate the integration job template to a separate file ci: Break off the integration_tests template into more templates ci: Introduce a template for upstream QEMU build ci: Add a new integration job template for the upstream QEMU scenario ci: Add a Fedora integration test job utilizing upstream QEMU ci/integration-template.yml | 98 +++++++++++++++++++++++++++++++++++++ ci/integration.yml | 70 +++++++------------------- 2 files changed, 116 insertions(+), 52 deletions(-) create mode 100644 ci/integration-template.yml -- 2.34.1
Future patches will do more code extraction to the template using YAML anchors so it's better that templates would live separately from the job definitions. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 52 +++++++++++++++++++++++++++++++++++ ci/integration.yml | 54 ++----------------------------------- 2 files changed, 54 insertions(+), 52 deletions(-) create mode 100644 ci/integration-template.yml diff --git a/ci/integration-template.yml b/ci/integration-template.yml new file mode 100644 index 0000000000..c4201df6db --- /dev/null +++ b/ci/integration-template.yml @@ -0,0 +1,52 @@ +.integration_tests: + stage: integration_tests + before_script: + - mkdir "$SCRATCH_DIR" + - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally + - sudo systemctl daemon-reexec # need to reexec systemd after changing config + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" = "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log"; + LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*"; + sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "$LOG_FILTERS" &>/dev/null; + sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "$LOG_OUTPUTS" &>/dev/null; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + - sudo virsh net-start default &>/dev/null || true; + script: + - mkdir logs + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git + - cd libvirt-tck + - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado + after_script: + - test "$CI_JOB_STATUS" = "success" && exit 0; + - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + - sudo coredumpctl info --no-pager > logs/coredumpctl.txt + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs + # rename all Avocado stderr/stdout logs to *.log so that GitLab's web UI doesn't mangle the MIME type + - find logs/avocado/ -type f ! -name "*.log" -exec + sh -c 'DIR=$(dirname {}); NAME=$(basename {}); mv $DIR/$NAME{,.log}' \; + variables: + SCRATCH_DIR: "/tmp/scratch" + artifacts: + name: logs + paths: + - logs + when: on_failure + rules: + - if: '$LIBVIRT_CI_INTEGRATION' + when: on_success + - when: never diff --git a/ci/integration.yml b/ci/integration.yml index 17f20b3945..a71df41694 100644 --- a/ci/integration.yml +++ b/ci/integration.yml @@ -1,55 +1,5 @@ -.integration_tests: - stage: integration_tests - before_script: - - mkdir "$SCRATCH_DIR" - - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally - - sudo systemctl daemon-reexec # need to reexec systemd after changing config - - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* - - sudo pip3 install --prefix=/usr avocado-framework - - source /etc/os-release # in order to query the vendor-provided variables - - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 || - test "$ID" = "fedora" && test "$VERSION_ID" -lt 35; - then - DAEMONS="libvirtd virtlogd virtlockd"; - else - DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; - fi - - for daemon in $DAEMONS; - do - LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log"; - LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*"; - sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "$LOG_FILTERS" &>/dev/null; - sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "$LOG_OUTPUTS" &>/dev/null; - sudo systemctl --quiet stop ${daemon}.service; - sudo systemctl restart ${daemon}.socket; - done - - sudo virsh net-start default &>/dev/null || true; - script: - - mkdir logs - - cd "$SCRATCH_DIR" - - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git - - cd libvirt-tck - - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado - after_script: - - test "$CI_JOB_STATUS" = "success" && exit 0; - - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; - - sudo coredumpctl info --no-pager > logs/coredumpctl.txt - - sudo mv /var/log/libvirt logs/libvirt - - sudo chown -R $(whoami):$(whoami) logs - # rename all Avocado stderr/stdout logs to *.log so that GitLab's web UI doesn't mangle the MIME type - - find logs/avocado/ -type f ! -name "*.log" -exec - sh -c 'DIR=$(dirname {}); NAME=$(basename {}); mv $DIR/$NAME{,.log}' \; - variables: - SCRATCH_DIR: "/tmp/scratch" - artifacts: - name: logs - paths: - - logs - when: on_failure - rules: - - if: '$LIBVIRT_CI_INTEGRATION' - when: on_success - - when: never +include: + - 'ci/integration-template.yml' centos-stream-8-tests: extends: .integration_tests -- 2.34.1
On Fri, May 06, 2022 at 05:35:16PM +0200, Erik Skultety wrote:
Future patches will do more code extraction to the template using YAML anchors so it's better that templates would live separately from the job definitions.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 52 +++++++++++++++++++++++++++++++++++ ci/integration.yml | 54 ++----------------------------------- 2 files changed, 54 insertions(+), 52 deletions(-) create mode 100644 ci/integration-template.yml
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
There's quite a lot happening int the .integration_tests template already even without adding upstream QEMU build to the mix. Let's break the template into more pieces which can be put together using the YAML anchors. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 76 +++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 29 deletions(-) diff --git a/ci/integration-template.yml b/ci/integration-template.yml index c4201df6db..195e9cfa5f 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,44 +1,62 @@ + + +.install-deps: &install-deps + - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* + - sudo pip3 install --prefix=/usr avocado-framework + + +.enable-core-dumps: &enable-core-dumps + - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally + - sudo systemctl daemon-reexec # need to reexec systemd after changing config + + +.enable-libvirt-debugging: &enable-libvirt-debugging + - source /etc/os-release # in order to query the vendor-provided variables + - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 || + test "$ID" = "fedora" && test "$VERSION_ID" -lt 35; + then + DAEMONS="libvirtd virtlogd virtlockd"; + else + DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; + fi + - for daemon in $DAEMONS; + do + LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log"; + LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*"; + sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "$LOG_FILTERS" &>/dev/null; + sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "$LOG_OUTPUTS" &>/dev/null; + sudo systemctl --quiet stop ${daemon}.service; + sudo systemctl restart ${daemon}.socket; + done + + +.collect-logs: &collect-logs + - mkdir logs + - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; + - sudo coredumpctl info --no-pager > logs/coredumpctl.txt + - sudo mv /var/log/libvirt logs/libvirt + - sudo chown -R $(whoami):$(whoami) logs + # rename all Avocado stderr/stdout logs to *.log so that GitLab's web UI doesn't mangle the MIME type + - find logs/avocado/ -type f ! -name "*.log" -exec + sh -c 'DIR=$(dirname {}); NAME=$(basename {}); mv $DIR/$NAME{,.log}' \; + + .integration_tests: stage: integration_tests before_script: - mkdir "$SCRATCH_DIR" - - sudo sh -c "echo DefaultLimitCORE=infinity >> /etc/systemd/system.conf" # Explicitly allow storing cores globally - - sudo systemctl daemon-reexec # need to reexec systemd after changing config - - sudo dnf install -y libvirt-rpms/* libvirt-perl-rpms/* - - sudo pip3 install --prefix=/usr avocado-framework - - source /etc/os-release # in order to query the vendor-provided variables - - if test "$ID" = "centos" && test "$VERSION_ID" -lt 9 || - test "$ID" = "fedora" && test "$VERSION_ID" -lt 35; - then - DAEMONS="libvirtd virtlogd virtlockd"; - else - DAEMONS="virtproxyd virtqemud virtinterfaced virtsecretd virtstoraged virtnwfilterd virtnodedevd virtlogd virtlockd"; - fi - - for daemon in $DAEMONS; - do - LOG_OUTPUTS="1:file:/var/log/libvirt/${daemon}.log"; - LOG_FILTERS="3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*"; - sudo augtool set /files/etc/libvirt/${daemon}.conf/log_filters "$LOG_FILTERS" &>/dev/null; - sudo augtool set /files/etc/libvirt/${daemon}.conf/log_outputs "$LOG_OUTPUTS" &>/dev/null; - sudo systemctl --quiet stop ${daemon}.service; - sudo systemctl restart ${daemon}.socket; - done + - *install-deps + - *enable-core-dumps + - *enable-libvirt-debugging - sudo virsh net-start default &>/dev/null || true; script: - - mkdir logs - cd "$SCRATCH_DIR" - git clone --depth 1 https://gitlab.com/libvirt/libvirt-tck.git - cd libvirt-tck - sudo avocado --config avocado.config run --job-results-dir "$SCRATCH_DIR"/avocado after_script: - test "$CI_JOB_STATUS" = "success" && exit 0; - - test -e "$SCRATCH_DIR"/avocado && sudo mv "$SCRATCH_DIR"/avocado/latest/test-results logs/avocado; - - sudo coredumpctl info --no-pager > logs/coredumpctl.txt - - sudo mv /var/log/libvirt logs/libvirt - - sudo chown -R $(whoami):$(whoami) logs - # rename all Avocado stderr/stdout logs to *.log so that GitLab's web UI doesn't mangle the MIME type - - find logs/avocado/ -type f ! -name "*.log" -exec - sh -c 'DIR=$(dirname {}); NAME=$(basename {}); mv $DIR/$NAME{,.log}' \; + - *collect-logs variables: SCRATCH_DIR: "/tmp/scratch" artifacts: -- 2.34.1
On Fri, May 06, 2022 at 05:35:17PM +0200, Erik Skultety wrote:
There's quite a lot happening int the .integration_tests template already even without adding upstream QEMU build to the mix. Let's break the template into more pieces which can be put together using the YAML anchors.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 76 +++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 29 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Rather than writing down the steps to build QEMU from source ourselves let's recycle what upstream QEMU uses in their CI. This has the obvious advantage that if QEMU changes something in their upstream CI build process we'll pull the change automatically. Since YAML anchors cannot be used in a cross-file scenario this patch makes use of the '!reference' keyword which allows us go even further and thus pick useful parts from the upstream QEMU's CI build template. In addition, we're also able to add more intermediary steps to these hand-picked QEMU build job sections. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/ci/integration-template.yml b/ci/integration-template.yml index 195e9cfa5f..d1934b1469 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,3 +1,20 @@ +# We want to reuse QEMU's .native_build_job_template to build it from source +include: + - project: 'qemu-project/qemu' + file: '.gitlab-ci.d/buildtest-template.yml' + + +.qemu-build-template: &qemu-build-template + - git clone --depth 1 https://gitlab.com/qemu-project/qemu.git + - cd qemu + # + # QEMU's .native_build_job_template bits + - export TARGETS="x86_64-softmmu" + - export CONFIGURE_ARGS="--prefix=/usr" + - export MAKE_CHECK_ARGS="check-build" + - !reference [.native_build_job_template, before_script] + - !reference [.native_build_job_template, script] + - sudo make install .install-deps: &install-deps -- 2.34.1
On Fri, May 06, 2022 at 05:35:18PM +0200, Erik Skultety wrote:
Rather than writing down the steps to build QEMU from source ourselves let's recycle what upstream QEMU uses in their CI. This has the obvious advantage that if QEMU changes something in their upstream CI build process we'll pull the change automatically. Since YAML anchors cannot be used in a cross-file scenario this patch makes use of the '!reference' keyword which allows us go even further and thus pick useful parts from the upstream QEMU's CI build template. In addition, we're also able to add more intermediary steps to these hand-picked QEMU build job sections.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index 195e9cfa5f..d1934b1469 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,3 +1,20 @@ +# We want to reuse QEMU's .native_build_job_template to build it from source +include: + - project: 'qemu-project/qemu' + file: '.gitlab-ci.d/buildtest-template.yml'
I don't think doing this is a good idea, because it is essentially a private impl detail of QEMU's current CI setup that can be changed at any time. Doing a simple QEMU build isn't complex, so I think we're fine to just put the logic in our code here directly.
+.qemu-build-template: &qemu-build-template + - git clone --depth 1 https://gitlab.com/qemu-project/qemu.git + - cd qemu + # + # QEMU's .native_build_job_template bits + - export TARGETS="x86_64-softmmu" + - export CONFIGURE_ARGS="--prefix=/usr" + - export MAKE_CHECK_ARGS="check-build" + - !reference [.native_build_job_template, before_script] + - !reference [.native_build_job_template, script] + - sudo make install
.install-deps: &install-deps -- 2.34.1
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 03:01:51PM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 05:35:18PM +0200, Erik Skultety wrote:
Rather than writing down the steps to build QEMU from source ourselves let's recycle what upstream QEMU uses in their CI. This has the obvious advantage that if QEMU changes something in their upstream CI build process we'll pull the change automatically. Since YAML anchors cannot be used in a cross-file scenario this patch makes use of the '!reference' keyword which allows us go even further and thus pick useful parts from the upstream QEMU's CI build template. In addition, we're also able to add more intermediary steps to these hand-picked QEMU build job sections.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index 195e9cfa5f..d1934b1469 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,3 +1,20 @@ +# We want to reuse QEMU's .native_build_job_template to build it from source +include: + - project: 'qemu-project/qemu' + file: '.gitlab-ci.d/buildtest-template.yml'
I don't think doing this is a good idea, because it is essentially a private impl detail of QEMU's current CI setup that can be changed at any time.
Doing a simple QEMU build isn't complex, so I think we're fine to just put the logic in our code here directly.
Well, sorry I already pushed the patches a day after mprivozn ACKed them. Besides, why is it bad that QEMU changes their build? Isn't it actually beneficial for us to build QEMU the exact same way as they do for integration purposes? I think that it alleviates any maintenance burden off our shoulders. Erik
+.qemu-build-template: &qemu-build-template + - git clone --depth 1 https://gitlab.com/qemu-project/qemu.git + - cd qemu + # + # QEMU's .native_build_job_template bits + - export TARGETS="x86_64-softmmu" + - export CONFIGURE_ARGS="--prefix=/usr" + - export MAKE_CHECK_ARGS="check-build" + - !reference [.native_build_job_template, before_script] + - !reference [.native_build_job_template, script] + - sudo make install
.install-deps: &install-deps -- 2.34.1
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 04:04:54PM +0200, Erik Skultety wrote:
On Wed, May 11, 2022 at 03:01:51PM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 05:35:18PM +0200, Erik Skultety wrote:
Rather than writing down the steps to build QEMU from source ourselves let's recycle what upstream QEMU uses in their CI. This has the obvious advantage that if QEMU changes something in their upstream CI build process we'll pull the change automatically. Since YAML anchors cannot be used in a cross-file scenario this patch makes use of the '!reference' keyword which allows us go even further and thus pick useful parts from the upstream QEMU's CI build template. In addition, we're also able to add more intermediary steps to these hand-picked QEMU build job sections.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index 195e9cfa5f..d1934b1469 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,3 +1,20 @@ +# We want to reuse QEMU's .native_build_job_template to build it from source +include: + - project: 'qemu-project/qemu' + file: '.gitlab-ci.d/buildtest-template.yml'
I don't think doing this is a good idea, because it is essentially a private impl detail of QEMU's current CI setup that can be changed at any time.
Doing a simple QEMU build isn't complex, so I think we're fine to just put the logic in our code here directly.
Well, sorry I already pushed the patches a day after mprivozn ACKed them.
Oh, I didn't see them in my local tree, so thought they weren't pushed yet.
Besides, why is it bad that QEMU changes their build? Isn't it actually beneficial for us to build QEMU the exact same way as they do for integration purposes? I think that it alleviates any maintenance burden off our shoulders.
I was refering to more to the existance of a 'buildtest-template.yml' file, or to the existance of specific build job names in this file, all of which have changed multiple times in QEMU. Conversely the actual way to invoke a QEMU build has not changed in the same timeframe. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 03:07:43PM +0100, Daniel P. Berrangé wrote:
On Wed, May 11, 2022 at 04:04:54PM +0200, Erik Skultety wrote:
On Wed, May 11, 2022 at 03:01:51PM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 05:35:18PM +0200, Erik Skultety wrote:
Rather than writing down the steps to build QEMU from source ourselves let's recycle what upstream QEMU uses in their CI. This has the obvious advantage that if QEMU changes something in their upstream CI build process we'll pull the change automatically. Since YAML anchors cannot be used in a cross-file scenario this patch makes use of the '!reference' keyword which allows us go even further and thus pick useful parts from the upstream QEMU's CI build template. In addition, we're also able to add more intermediary steps to these hand-picked QEMU build job sections.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index 195e9cfa5f..d1934b1469 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,3 +1,20 @@ +# We want to reuse QEMU's .native_build_job_template to build it from source +include: + - project: 'qemu-project/qemu' + file: '.gitlab-ci.d/buildtest-template.yml'
I don't think doing this is a good idea, because it is essentially a private impl detail of QEMU's current CI setup that can be changed at any time.
Doing a simple QEMU build isn't complex, so I think we're fine to just put the logic in our code here directly.
Well, sorry I already pushed the patches a day after mprivozn ACKed them.
Oh, I didn't see them in my local tree, so thought they weren't pushed yet.
Are you sure they are pushed ? There's no job appearing: https://gitlab.com/libvirt/libvirt/-/pipelines/536112780 and I'm not seeing them after fetching latest origin/master With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 03:11:01PM +0100, Daniel P. Berrangé wrote:
On Wed, May 11, 2022 at 03:07:43PM +0100, Daniel P. Berrangé wrote:
On Wed, May 11, 2022 at 04:04:54PM +0200, Erik Skultety wrote:
On Wed, May 11, 2022 at 03:01:51PM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 05:35:18PM +0200, Erik Skultety wrote:
Rather than writing down the steps to build QEMU from source ourselves let's recycle what upstream QEMU uses in their CI. This has the obvious advantage that if QEMU changes something in their upstream CI build process we'll pull the change automatically. Since YAML anchors cannot be used in a cross-file scenario this patch makes use of the '!reference' keyword which allows us go even further and thus pick useful parts from the upstream QEMU's CI build template. In addition, we're also able to add more intermediary steps to these hand-picked QEMU build job sections.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index 195e9cfa5f..d1934b1469 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -1,3 +1,20 @@ +# We want to reuse QEMU's .native_build_job_template to build it from source +include: + - project: 'qemu-project/qemu' + file: '.gitlab-ci.d/buildtest-template.yml'
I don't think doing this is a good idea, because it is essentially a private impl detail of QEMU's current CI setup that can be changed at any time.
Doing a simple QEMU build isn't complex, so I think we're fine to just put the logic in our code here directly.
Well, sorry I already pushed the patches a day after mprivozn ACKed them.
Oh, I didn't see them in my local tree, so thought they weren't pushed yet.
Are you sure they are pushed ? There's no job appearing:
https://gitlab.com/libvirt/libvirt/-/pipelines/536112780
and I'm not seeing them after fetching latest origin/master
Doh! Indeed they are not. I remember merging them to my local master and then a meeting distracted me from pushing them and then I clearly forgot to do it, but my head convinced me that I had done it :). Alright then, let me copy-paste the QEMU build job template to libvirt instead of pulling it. Erik
YAML anchors don't work with Shell conditions structures, so we cannot simply pull in the QEMU build template conditionally like we do for the other anchors. Instead we have to 'subclass' the .integration_tests template and inject the QEMU building bits. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/ci/integration-template.yml b/ci/integration-template.yml index d1934b1469..3bad143622 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -85,3 +85,14 @@ include: - if: '$LIBVIRT_CI_INTEGRATION' when: on_success - when: never + + +# YAML anchors don't work with Shell conditions so we can't use a variable +# to conditionally build+install QEMU from source. +# Instead, create a new test job template for this scenario. +.integration_tests_upstream_qemu: + extends: .integration_tests + before_script: + - !reference [.integration_tests, before_script] + - cd "$SCRATCH_DIR" + - *qemu-build-template -- 2.34.1
On Fri, May 06, 2022 at 05:35:19PM +0200, Erik Skultety wrote:
YAML anchors don't work with Shell conditions structures, so we cannot simply pull in the QEMU build template conditionally like we do for the other anchors. Instead we have to 'subclass' the .integration_tests template and inject the QEMU building bits.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 11 +++++++++++ 1 file changed, 11 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index d1934b1469..3bad143622 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -85,3 +85,14 @@ include: - if: '$LIBVIRT_CI_INTEGRATION' when: on_success - when: never + + +# YAML anchors don't work with Shell conditions so we can't use a variable +# to conditionally build+install QEMU from source. +# Instead, create a new test job template for this scenario. +.integration_tests_upstream_qemu: + extends: .integration_tests + before_script: + - !reference [.integration_tests, before_script] + - cd "$SCRATCH_DIR" + - *qemu-build-template
If we're not pull in QEMU's external gitlab config, them presumably this problem won't exist anymore. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 03:03:18PM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 05:35:19PM +0200, Erik Skultety wrote:
YAML anchors don't work with Shell conditions structures, so we cannot simply pull in the QEMU build template conditionally like we do for the other anchors. Instead we have to 'subclass' the .integration_tests template and inject the QEMU building bits.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 11 +++++++++++ 1 file changed, 11 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index d1934b1469..3bad143622 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -85,3 +85,14 @@ include: - if: '$LIBVIRT_CI_INTEGRATION' when: on_success - when: never + + +# YAML anchors don't work with Shell conditions so we can't use a variable +# to conditionally build+install QEMU from source. +# Instead, create a new test job template for this scenario. +.integration_tests_upstream_qemu: + extends: .integration_tests + before_script: + - !reference [.integration_tests, before_script] + - cd "$SCRATCH_DIR" + - *qemu-build-template
If we're not pull in QEMU's external gitlab config, them presumably this problem won't exist anymore.
It will because either way you can't do something like ''' script: - if $VAR; then *template fi ''' so the original integration_tests template has to be further extended or the QEMU build needs to be opencoded inside with the latter being an inferior approach IMO. Erik
On Wed, May 11, 2022 at 04:36:48PM +0200, Erik Skultety wrote:
On Wed, May 11, 2022 at 03:03:18PM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 05:35:19PM +0200, Erik Skultety wrote:
YAML anchors don't work with Shell conditions structures, so we cannot simply pull in the QEMU build template conditionally like we do for the other anchors. Instead we have to 'subclass' the .integration_tests template and inject the QEMU building bits.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration-template.yml | 11 +++++++++++ 1 file changed, 11 insertions(+)
diff --git a/ci/integration-template.yml b/ci/integration-template.yml index d1934b1469..3bad143622 100644 --- a/ci/integration-template.yml +++ b/ci/integration-template.yml @@ -85,3 +85,14 @@ include: - if: '$LIBVIRT_CI_INTEGRATION' when: on_success - when: never + + +# YAML anchors don't work with Shell conditions so we can't use a variable +# to conditionally build+install QEMU from source. +# Instead, create a new test job template for this scenario. +.integration_tests_upstream_qemu: + extends: .integration_tests + before_script: + - !reference [.integration_tests, before_script] + - cd "$SCRATCH_DIR" + - *qemu-build-template
If we're not pull in QEMU's external gitlab config, them presumably this problem won't exist anymore.
It will because either way you can't do something like
''' script: - if $VAR; then *template fi '''
so the original integration_tests template has to be further extended or the QEMU build needs to be opencoded inside with the latter being an inferior approach IMO.
Ah, I see what you mean now. In that case Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/ci/integration.yml b/ci/integration.yml index a71df41694..4978d3cf02 100644 --- a/ci/integration.yml +++ b/ci/integration.yml @@ -64,3 +64,19 @@ fedora-35-tests: LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host tags: - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG + +fedora-35-upstream-qemu-tests: + extends: .integration_tests_upstream_qemu + needs: + - x86_64-fedora-35 + - project: libvirt/libvirt-perl + job: x86_64-fedora-35 + ref: master + artifacts: true + variables: + # needed by libvirt-gitlab-executor + DISTRO: fedora-35 + # can be overridden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: redhat-vm-host + tags: + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG -- 2.34.1
On Fri, May 06, 2022 at 05:35:20PM +0200, Erik Skultety wrote:
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- ci/integration.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, May 06, 2022 at 05:35:15PM +0200, Erik Skultety wrote:
Since QEMU doesn't maintain a spec file in upstream, we cannot build RPM artifacts as part of the CI as we do for libvirt. Instead of hard-coding the build steps for QEMU though patch 3/5 pulls in QEMU's CI job template which means we'll remain in sync if QEMU makes changes to its build process.
Oh, I forgot to attach a successful CI demo pipeline: https://gitlab.com/eskultety/libvirt/-/pipelines/533060438
On 5/6/22 17:35, Erik Skultety wrote:
Since QEMU doesn't maintain a spec file in upstream, we cannot build RPM artifacts as part of the CI as we do for libvirt. Instead of hard-coding the build steps for QEMU though patch 3/5 pulls in QEMU's CI job template which means we'll remain in sync if QEMU makes changes to its build process.
Erik Skultety (5): ci: Separate the integration job template to a separate file ci: Break off the integration_tests template into more templates ci: Introduce a template for upstream QEMU build ci: Add a new integration job template for the upstream QEMU scenario ci: Add a Fedora integration test job utilizing upstream QEMU
ci/integration-template.yml | 98 +++++++++++++++++++++++++++++++++++++ ci/integration.yml | 70 +++++++------------------- 2 files changed, 116 insertions(+), 52 deletions(-) create mode 100644 ci/integration-template.yml
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Michal
participants (3)
-
Daniel P. Berrangé -
Erik Skultety -
Michal Prívozník