[libvirt PATCH 0/2] gitlab: Clean up and improve ccache usage
Test pipeline: https://gitlab.com/abologna/libvirt/pipelines/131128028 Andrea Bolognani (2): gitlab: Don't define $MAKE gitlab: Enable improved ccache usage .gitlab-ci.yml | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) -- 2.25.1
Since libvirt-jenkins-ci commit 27cfddee8835, paths to build tools such as ninja and make are exported in the container's environment and can be used directly. Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index beeae1df05..3203434b99 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,5 +1,4 @@ variables: - MAKE: make GIT_DEPTH: 100 stages: -- 2.25.1
On Mon, Mar 30, 2020 at 07:31:02PM +0200, Andrea Bolognani wrote:
Since libvirt-jenkins-ci commit 27cfddee8835, paths to build tools such as ninja and make are exported in the container's environment and can be used directly.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 1 - 1 file changed, 1 deletion(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Setting CC="ccache cc" works in most cases, but sometimes it will break the build: in particular, we have experienced issues in the past with that approach when using cgo to build our Go bindings. A more robust approach is to have a directory containing symlinks from the compiler name to the ccache binary: in that case, ccache itself will invoke the compiler, and the build system will be none the wiser. Since libvirt-jenkins-ci commit 2563aebb6c5c, container images contain a suitable symlink directory, so all that's needed to enable the new approach is to add this directory to $PATH. Since we're touching this anyway, we make a few more changes: $CCACHE_DIR is no longer created manually, because ccache will take care of creating it for us if it doesn't already exist; the ccache setup is moved out of the job template and into script_variables, removing unnecessary duplication; a limit is set on the size of the cache (500 MB, which is twice the amount used by a fresh build on my Fedora 31 machine). Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 3203434b99..99e7b510c7 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -8,6 +8,10 @@ stages: .script_variables: &script_variables | export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + export CCACHE_BASEDIR="$(pwd)" + export CCACHE_DIR="$CCACHE_BASEDIR/ccache" + export CCACHE_MAXSIZE="500M" + export PATH="$CCACHE_WRAPPERSDIR:$PATH" # Common templates @@ -20,10 +24,6 @@ stages: key: "$CI_JOB_NAME" before_script: - *script_variables - - mkdir -p ccache - - export CC="ccache gcc" - - export CCACHE_BASEDIR=${PWD} - - export CCACHE_DIR=${PWD}/ccache script: - mkdir build - cd build @@ -48,10 +48,6 @@ stages: key: "$CI_JOB_NAME" before_script: - *script_variables - - mkdir -p ccache - - export CC="ccache ${ABI}-gcc" - - export CCACHE_BASEDIR=${PWD} - - export CCACHE_DIR=${PWD}/ccache script: - mkdir build - cd build -- 2.25.1
On Mon, Mar 30, 2020 at 07:31:03PM +0200, Andrea Bolognani wrote:
Setting CC="ccache cc" works in most cases, but sometimes it will break the build: in particular, we have experienced issues in the past with that approach when using cgo to build our Go bindings.
A more robust approach is to have a directory containing symlinks from the compiler name to the ccache binary: in that case, ccache itself will invoke the compiler, and the build system will be none the wiser.
Since libvirt-jenkins-ci commit 2563aebb6c5c, container images contain a suitable symlink directory, so all that's needed to enable the new approach is to add this directory to $PATH.
Since we're touching this anyway, we make a few more changes: $CCACHE_DIR is no longer created manually, because ccache will take care of creating it for us if it doesn't already exist; the
I could have sworn I had a build failure when I didn't have the mkdir present when originally doing this recipe. It might have been from gitlab itself, rather than ccache, eg when unpacking the cache archive to the non-existant dir. But assuming the CI pipeline works for you...
ccache setup is moved out of the job template and into script_variables, removing unnecessary duplication; a limit is set on the size of the cache (500 MB, which is twice the amount used by a fresh build on my Fedora 31 machine).
Signed-off-by: Andrea Bolognani <abologna@redhat.com> --- .gitlab-ci.yml | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, 2020-04-15 at 13:38 +0100, Daniel P. Berrangé wrote:
On Mon, Mar 30, 2020 at 07:31:03PM +0200, Andrea Bolognani wrote:
Since we're touching this anyway, we make a few more changes: $CCACHE_DIR is no longer created manually, because ccache will take care of creating it for us if it doesn't already exist; the
I could have sworn I had a build failure when I didn't have the mkdir present when originally doing this recipe. It might have been from gitlab itself, rather than ccache, eg when unpacking the cache archive to the non-existant dir. But assuming the CI pipeline works for you...
I tried again to make sure, because I had not considered the GitLab angle. If you look at any of the build on https://gitlab.com/abologna/libvirt/pipelines/136291992 you'll see that the cache gets restored successfully. -- Andrea Bolognani / Red Hat / Virtualization
On Mon, 2020-03-30 at 19:31 +0200, Andrea Bolognani wrote:
Test pipeline: https://gitlab.com/abologna/libvirt/pipelines/131128028
Andrea Bolognani (2): gitlab: Don't define $MAKE gitlab: Enable improved ccache usage
.gitlab-ci.yml | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-)
alias ping="ccache ping" -- Andrea Bolognani / Red Hat / Virtualization
participants (2)
-
Andrea Bolognani -
Daniel P. Berrangé