[libvirt] [jenkins-ci PATCH] Use shallow git clones for projects
First of all, we don't need the full history for builds and we can save some time during the prepare phase. The disk footprint is about 5x smaller, but the most important thing is the repo clone duration difference e.g. in libvirt's case is quite significant: full history: real 3m45.236s user 1m53.074s sys 0m15.577s depth 1: real 0m14.868s user 0m2.215s sys 0m0.906s --- guests/playbooks/build/jobs/prepare.yml | 1 + jenkins/jobs/autotools.yaml | 1 + jenkins/jobs/generic.yaml | 1 + jenkins/jobs/go.yaml | 1 + jenkins/jobs/perl-modulebuild.yaml | 1 + jenkins/jobs/python-distutils.yaml | 1 + 6 files changed, 6 insertions(+) diff --git a/guests/playbooks/build/jobs/prepare.yml b/guests/playbooks/build/jobs/prepare.yml index 576afbc..35ca9e4 100644 --- a/guests/playbooks/build/jobs/prepare.yml +++ b/guests/playbooks/build/jobs/prepare.yml @@ -5,6 +5,7 @@ version: '{{ git_branch }}' dest: '{{ name }}' force: yes + depth: 1 when: - inventory_hostname in machines diff --git a/jenkins/jobs/autotools.yaml b/jenkins/jobs/autotools.yaml index f04ec17..1a85781 100644 --- a/jenkins/jobs/autotools.yaml +++ b/jenkins/jobs/autotools.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/generic.yaml b/jenkins/jobs/generic.yaml index ca69513..f58945e 100644 --- a/jenkins/jobs/generic.yaml +++ b/jenkins/jobs/generic.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/go.yaml b/jenkins/jobs/go.yaml index 1bdcb5f..dd3c5d4 100644 --- a/jenkins/jobs/go.yaml +++ b/jenkins/jobs/go.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/perl-modulebuild.yaml b/jenkins/jobs/perl-modulebuild.yaml index 7173493..0ea271e 100644 --- a/jenkins/jobs/perl-modulebuild.yaml +++ b/jenkins/jobs/perl-modulebuild.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/python-distutils.yaml b/jenkins/jobs/python-distutils.yaml index f56794c..a3eb076 100644 --- a/jenkins/jobs/python-distutils.yaml +++ b/jenkins/jobs/python-distutils.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' -- 2.20.1
On Tue, Aug 27, 2019 at 02:56:02PM +0200, Erik Skultety wrote:
First of all, we don't need the full history for builds and we can save some time during the prepare phase. The disk footprint is about 5x smaller, but the most important thing is the repo clone duration difference e.g. in libvirt's case is quite significant:
full history: real 3m45.236s user 1m53.074s sys 0m15.577s
depth 1: real 0m14.868s user 0m2.215s sys 0m0.906s
On Jenkins, the perf diff should be negligible. The CI build host already has a full git clone with entire history. So all the worker VMs are effectively cloning from localhost, not hitting the WAN. They also are never doing a full git clone, just a git pull to get the delta from the previous run. Personally I find it useful to have the history when I'm debugging problems on my build host/VM, so I'm not really in favour of doing this change. It would make the VMs setup less usable for me, so I prefer to accept the time penalty. Is there something we can do to make the standalone VMs pull from a repo on the build host by default, instead of fetching from libvirt.org. Or to get their initial clone from the host, but then get subsequent incremental updates from libvirt.org ?
--- guests/playbooks/build/jobs/prepare.yml | 1 + jenkins/jobs/autotools.yaml | 1 + jenkins/jobs/generic.yaml | 1 + jenkins/jobs/go.yaml | 1 + jenkins/jobs/perl-modulebuild.yaml | 1 + jenkins/jobs/python-distutils.yaml | 1 + 6 files changed, 6 insertions(+)
diff --git a/guests/playbooks/build/jobs/prepare.yml b/guests/playbooks/build/jobs/prepare.yml index 576afbc..35ca9e4 100644 --- a/guests/playbooks/build/jobs/prepare.yml +++ b/guests/playbooks/build/jobs/prepare.yml @@ -5,6 +5,7 @@ version: '{{ git_branch }}' dest: '{{ name }}' force: yes + depth: 1 when: - inventory_hostname in machines
diff --git a/jenkins/jobs/autotools.yaml b/jenkins/jobs/autotools.yaml index f04ec17..1a85781 100644 --- a/jenkins/jobs/autotools.yaml +++ b/jenkins/jobs/autotools.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/generic.yaml b/jenkins/jobs/generic.yaml index ca69513..f58945e 100644 --- a/jenkins/jobs/generic.yaml +++ b/jenkins/jobs/generic.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/go.yaml b/jenkins/jobs/go.yaml index 1bdcb5f..dd3c5d4 100644 --- a/jenkins/jobs/go.yaml +++ b/jenkins/jobs/go.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/perl-modulebuild.yaml b/jenkins/jobs/perl-modulebuild.yaml index 7173493..0ea271e 100644 --- a/jenkins/jobs/perl-modulebuild.yaml +++ b/jenkins/jobs/perl-modulebuild.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}' diff --git a/jenkins/jobs/python-distutils.yaml b/jenkins/jobs/python-distutils.yaml index f56794c..a3eb076 100644 --- a/jenkins/jobs/python-distutils.yaml +++ b/jenkins/jobs/python-distutils.yaml @@ -27,6 +27,7 @@ after: true skip-tag: true wipe-workspace: false + shallow-clone: true triggers: - reverse: jobs: '{obj:parent_jobs}'
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Aug 27, 2019 at 02:12:12PM +0100, Daniel P. Berrangé wrote:
On Tue, Aug 27, 2019 at 02:56:02PM +0200, Erik Skultety wrote:
First of all, we don't need the full history for builds and we can save some time during the prepare phase. The disk footprint is about 5x smaller, but the most important thing is the repo clone duration difference e.g. in libvirt's case is quite significant:
full history: real 3m45.236s user 1m53.074s sys 0m15.577s
depth 1: real 0m14.868s user 0m2.215s sys 0m0.906s
On Jenkins, the perf diff should be negligible. The CI build host already has a full git clone with entire history. So all the worker VMs are effectively cloning from localhost, not hitting the WAN. They also are never doing a full git clone, just a git pull to get the delta from the previous run.
Personally I find it useful to have the history when I'm debugging problems on my build host/VM, so I'm not really in favour of doing this change. It would make the VMs setup less usable for me, so I prefer to accept the time penalty.
Hmm, I have often machines that come and go, could we make this a variable then? If set we'd make a shallow copy and by default leave it 'false' for the local CI build host and drop the jenkins changes?
Is there something we can do to make the standalone VMs pull from a repo on the build host by default, instead of fetching from libvirt.org. Or to get their initial clone from the host, but then get subsequent incremental updates from libvirt.org ?
For that, you need to change the git_url in guests/playbooks/build/jobs/defaults.yml locally, not something we can make universal for everyone. Erik
On Tue, Aug 27, 2019 at 03:39:53PM +0200, Erik Skultety wrote:
On Tue, Aug 27, 2019 at 02:12:12PM +0100, Daniel P. Berrangé wrote:
On Tue, Aug 27, 2019 at 02:56:02PM +0200, Erik Skultety wrote:
First of all, we don't need the full history for builds and we can save some time during the prepare phase. The disk footprint is about 5x smaller, but the most important thing is the repo clone duration difference e.g. in libvirt's case is quite significant:
full history: real 3m45.236s user 1m53.074s sys 0m15.577s
depth 1: real 0m14.868s user 0m2.215s sys 0m0.906s
On Jenkins, the perf diff should be negligible. The CI build host already has a full git clone with entire history. So all the worker VMs are effectively cloning from localhost, not hitting the WAN. They also are never doing a full git clone, just a git pull to get the delta from the previous run.
Personally I find it useful to have the history when I'm debugging problems on my build host/VM, so I'm not really in favour of doing this change. It would make the VMs setup less usable for me, so I prefer to accept the time penalty.
Hmm, I have often machines that come and go, could we make this a variable then? If set we'd make a shallow copy and by default leave it 'false' for the local CI build host and drop the jenkins changes?
Yeah that sounds ok Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (2)
-
Daniel P. Berrangé -
Erik Skultety