10:05 a.m.
For macOS you always get the maximum configuration by default (12 CPUs,
24 GB RAM), but for FreeBSD you get 2 CPUs, 4 GBs by default. This
change increases the allocation to 8 CPUs, 8 GBs for FreeBSD.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
In theory this could make builds quicker. In practice I've not been
able to measure a difference due to large variance between runs.
.gitlab-ci.yml | 8 ++++++++
ci/cirrus/build.yml | 2 ++
2 files changed, 10 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3cb6ff5e6b..24588628f2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -114,6 +114,8 @@ stages:
-e "s|[@]CIRRUS_VM_INSTANCE_TYPE@|$CIRRUS_VM_INSTANCE_TYPE|g"
-e "s|[@]CIRRUS_VM_IMAGE_SELECTOR@|$CIRRUS_VM_IMAGE_SELECTOR|g"
-e "s|[@]CIRRUS_VM_IMAGE_NAME@|$CIRRUS_VM_IMAGE_NAME|g"
+ -e "s|[@]CIRRUS_VM_CPUS@|$CIRRUS_VM_CPUS|g"
+ -e "s|[@]CIRRUS_VM_RAM@|$CIRRUS_VM_RAM|g"
-e "s|[@]UPDATE_COMMAND@|$UPDATE_COMMAND|g"
-e "s|[@]UPGRADE_COMMAND@|$UPGRADE_COMMAND|g"
-e "s|[@]INSTALL_COMMAND@|$INSTALL_COMMAND|g"
@@ -423,6 +425,8 @@ x64-freebsd-12-build:
CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
CIRRUS_VM_IMAGE_SELECTOR: image_family
CIRRUS_VM_IMAGE_NAME: freebsd-12-2
+ CIRRUS_VM_CPUS: 8
+ CIRRUS_VM_RAM: 8G
UPDATE_COMMAND: pkg update
UPGRADE_COMMAND: pkg upgrade -y
INSTALL_COMMAND: pkg install -y
@@ -434,6 +438,8 @@ x64-freebsd-13-build:
CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
CIRRUS_VM_IMAGE_SELECTOR: image_family
CIRRUS_VM_IMAGE_NAME: freebsd-13-0
+ CIRRUS_VM_CPUS: 8
+ CIRRUS_VM_RAM: 8G
UPDATE_COMMAND: pkg update
UPGRADE_COMMAND: pkg upgrade -y
INSTALL_COMMAND: pkg install -y
@@ -445,6 +451,8 @@ x64-macos-11-build:
CIRRUS_VM_INSTANCE_TYPE: osx_instance
CIRRUS_VM_IMAGE_SELECTOR: image
CIRRUS_VM_IMAGE_NAME: big-sur-base
+ CIRRUS_VM_CPUS: 12
+ CIRRUS_VM_RAM: 24G
UPDATE_COMMAND: brew update
UPGRADE_COMMAND: brew upgrade
INSTALL_COMMAND: brew install
diff --git a/ci/cirrus/build.yml b/ci/cirrus/build.yml
index 867d5f297b..e9ad427765 100644
--- a/ci/cirrus/build.yml
+++ b/ci/cirrus/build.yml
@@ -1,5 +1,7 @@
@CIRRUS_VM_INSTANCE_TYPE@:
@CIRRUS_VM_IMAGE_SELECTOR@: @CIRRUS_VM_IMAGE_NAME@
+ cpu: @CIRRUS_VM_CPUS@
+ memory: @CIRRUS_VM_RAM@
env:
CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
--
2.31.1
5:39 a.m.
On Tue, Jul 20, 2021 at 04:05:22PM +0100, Daniel P. Berrangé wrote:
For macOS you always get the maximum configuration by default (12
CPUs,
24 GB RAM), but for FreeBSD you get 2 CPUs, 4 GBs by default. This
change increases the allocation to 8 CPUs, 8 GBs for FreeBSD.
---
In theory this could make builds quicker. In practice I've not been
able to measure a difference due to large variance between runs.
Should we actually do this? The fact that FreeBSD builds take so much
time is painful, but if you haven't been able to measure significant
improvements from maximizing the number of CPUs assigned to the VM
then perhaps other factors such as slow disk speed are to blame
instead. In fact, looking at recent jobs it looks like they mostly
fall in line with Linux builds...
We certainly don't need 24 GiB of RAM to build libvirt, and grabbing
12 CPUs seems excessively greedy too. Can we try being nice citizens
instead, and allocate something like 4 CPUs and 2 GiB of RAM across
the board? I expect that the resulting build times would still be
close to what we're seeing for containerized Linux builds, and so the
overall pipeline completion time won't be negatively affected.
--
Andrea Bolognani / Red Hat / Virtualization
5:48 a.m.
On Wed, Jul 21, 2021 at 06:39:07AM -0400, Andrea Bolognani wrote:
On Tue, Jul 20, 2021 at 04:05:22PM +0100, Daniel P. Berrangé wrote:
> For macOS you always get the maximum configuration by default (12 CPUs,
> 24 GB RAM), but for FreeBSD you get 2 CPUs, 4 GBs by default. This
> change increases the allocation to 8 CPUs, 8 GBs for FreeBSD.
>
> ---
> In theory this could make builds quicker. In practice I've not been
> able to measure a difference due to large variance between runs.
Should we actually do this? The fact that FreeBSD builds take so much
time is painful, but if you haven't been able to measure significant
improvements from maximizing the number of CPUs assigned to the VM
then perhaps other factors such as slow disk speed are to blame
instead. In fact, looking at recent jobs it looks like they mostly
fall in line with Linux builds...
Yeah, I'm in two minds and just sent this patch to get a second
opinion.
I noticed the ability to configure this when switching QEMU to
use lcitool for generating the freebsd/macos configuration.
The increase in resources for FreeBSD VMs made a significant
difference to QEMU build times - the difference between passing
and hitting the job limit timeout.
Thus I'm really puzzelled why libvirt didn't see a real benefit
when I tested it. As you say though, FreeBSD builds are no worse
than Linux builds, so it isn't critical for us.
We certainly don't need 24 GiB of RAM to build libvirt, and
grabbing
12 CPUs seems excessively greedy too. Can we try being nice citizens
instead, and allocate something like 4 CPUs and 2 GiB of RAM across
the board? I expect that the resulting build times would still be
close to what we're seeing for containerized Linux builds, and so the
overall pipeline completion time won't be negatively affected.
FWIW, for macOS systems you get the maximum resources no matter what,
as they always allocate you an entire system. I presume this is
because macOS builders run on physical hardware due to macOS license
rules preventing VM usages.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
6:06 a.m.
On Wed, Jul 21, 2021 at 11:48:17AM +0100, Daniel P. Berrangé wrote:
On Wed, Jul 21, 2021 at 06:39:07AM -0400, Andrea Bolognani wrote:
> On Tue, Jul 20, 2021 at 04:05:22PM +0100, Daniel P. Berrangé wrote:
> > For macOS you always get the maximum configuration by default (12 CPUs,
> > 24 GB RAM), but for FreeBSD you get 2 CPUs, 4 GBs by default. This
> > change increases the allocation to 8 CPUs, 8 GBs for FreeBSD.
> >
> > ---
> > In theory this could make builds quicker. In practice I've not been
> > able to measure a difference due to large variance between runs.
>
> Should we actually do this? The fact that FreeBSD builds take so much
> time is painful, but if you haven't been able to measure significant
> improvements from maximizing the number of CPUs assigned to the VM
> then perhaps other factors such as slow disk speed are to blame
> instead. In fact, looking at recent jobs it looks like they mostly
> fall in line with Linux builds...
Yeah, I'm in two minds and just sent this patch to get a second
opinion.
I noticed the ability to configure this when switching QEMU to
use lcitool for generating the freebsd/macos configuration.
The increase in resources for FreeBSD VMs made a significant
difference to QEMU build times - the difference between passing
and hitting the job limit timeout.
Thus I'm really puzzelled why libvirt didn't see a real benefit
when I tested it. As you say though, FreeBSD builds are no worse
than Linux builds, so it isn't critical for us.
I guess looking at the CPU chart in this run explains it:
https://cirrus-ci.com/task/5496170802839552
The first 75% of the job time was using less than 1 CPU. Only
the last 3 minutes maxed out both CPUs. So adding more CPUS
will only help the last bit of the job and even then we'll
hit limits on amount of parallelism due to build dependancies.
QEMU simply has much much more stuff needing building and is
highly parallel when building, so saw a big benefit.
So this affirms that we dont really need this patch.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:14 a.m.
On Wed, Jul 21, 2021 at 12:06:35PM +0100, Daniel P. Berrangé wrote:
On Wed, Jul 21, 2021 at 11:48:17AM +0100, Daniel P. Berrangé wrote:
> Thus I'm really puzzelled why libvirt didn't see a real benefit
> when I tested it. As you say though, FreeBSD builds are no worse
> than Linux builds, so it isn't critical for us.
I guess looking at the CPU chart in this run explains it:
https://cirrus-ci.com/task/5496170802839552
The first 75% of the job time was using less than 1 CPU. Only
the last 3 minutes maxed out both CPUs. So adding more CPUS
will only help the last bit of the job and even then we'll
hit limits on amount of parallelism due to build dependancies.
QEMU simply has much much more stuff needing building and is
highly parallel when building, so saw a big benefit.
So this affirms that we dont really need this patch.
We could still spell out our expectations in term of resource
allocation, and just set 4 CPUs / 2 GiB of RAM across the board. I
think that would be still moderately useful, for documentation
purposes if nothing else, and might help Cirrus CI better distribute
its computing resources among users.
We don't even need to make it dynamic per job, we can just hardcode
those values in ci/cirrus/build.yml and call it a day.
--
Andrea Bolognani / Red Hat / Virtualization
6:07 a.m.
On 7/20/21 5:05 PM, Daniel P. Berrangé wrote:
For macOS you always get the maximum configuration by default (12
CPUs,
24 GB RAM), but for FreeBSD you get 2 CPUs, 4 GBs by default. This
change increases the allocation to 8 CPUs, 8 GBs for FreeBSD.
Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
---
In theory this could make builds quicker. In practice I've not been
able to measure a difference due to large variance between runs.
.gitlab-ci.yml | 8 ++++++++
ci/cirrus/build.yml | 2 ++
2 files changed, 10 insertions(+)
A slightly offtopic question - is there a limit how many builds can we
run in given time frame, or how much resources can we consume? If so,
how far are we from getting rate limited? My understanding is that we
are using a free plan and I bet there would be some sort of limit.
Michal
6:09 a.m.
On Wed, Jul 21, 2021 at 01:07:04PM +0200, Michal Prívozník wrote:
On 7/20/21 5:05 PM, Daniel P. Berrangé wrote:
> For macOS you always get the maximum configuration by default (12 CPUs,
> 24 GB RAM), but for FreeBSD you get 2 CPUs, 4 GBs by default. This
> change increases the allocation to 8 CPUs, 8 GBs for FreeBSD.
>
> Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com>
> ---
>
> In theory this could make builds quicker. In practice I've not been
> able to measure a difference due to large variance between runs.
>
> .gitlab-ci.yml | 8 ++++++++
> ci/cirrus/build.yml | 2 ++
> 2 files changed, 10 insertions(+)
A slightly offtopic question - is there a limit how many builds can we
run in given time frame, or how much resources can we consume? If so,
how far are we from getting rate limited? My understanding is that we
are using a free plan and I bet there would be some sort of limit.
Cirrus doesn't limit overall consumption of resources, but it does
limit job parallelism. IIRC, you can only have 2 jobs runing at any
point in time in Cirrus. So we need to keep the number of jobs low.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1273
days inactive
1274
days old
6 comments
3 participants
participants (3)
-
Andrea Bolognani -
Daniel P. Berrangé -
Michal Prívozník