Re: [libvirt] [jenkins-ci PATCH 0/1] jenkins: Start building on Ubuntu
On Tue, Dec 10, 2019 at 02:54:22PM +0100, Andrea Bolognani wrote:
This patch is intended to start a slightly larger discussion about
our plans for the CentOS CI environment going forward.
At the moment, we have active builders for
CentOS 7
Debian 9
Debian 10
Fedora 30
Fedora 31
Fedora Rawhide
FreeBSD 11
FreeBSD 12
but we don't have builder for
Debian sid
FreeBSD -CURRENT
Ubuntu 16.04
Ubuntu 18.04
despite them being fully supported in the libvirt-jenkins-ci
repository.
This makes sense for sid and -CURRENT, since the former covers the
same "freshest Linux packages" angle that Rawhide already takes care
of and the latter is often broken and not trivial to keep updated;
both Ubuntu targets, however, should IMHO be part of the CentOS CI
environment. Hence this series :)
Moreover, we're in the process of adding
CentOS 8
openSUSE Leap 15.1
openSUSE Tumbleweed
as targets, of which the first two should also IMHO be added as they
would provide useful additional coverage.
The only reason why I'm even questioning whether this should be done
is capacity for the hypervisor host: the machine we're running all
builders on has
CPUs: 8
Memory: 32 GiB
Storage: 450 GiB
and each of the guests is configured to use
CPUs: 2
Memory: 2 GiB
Storage: 20 GiB
So while we're good, and actually have plenty of room to grow, on
the memory and storage front, we're already overcommitting our CPUs
pretty significantly, which I guess is at least part of the reason
why builds take so long.
NB the memory that's free is not really free - it is being usefull
as I/O cache for the VM disks. So more VMs will reduce I/O cache.
Whether that will actually impact us I don't know though.
More importantly though, AFAICT, those are not 8 real CPUs.
virsh nodeinfo reports 8 cores, but virsh capabilities
reports it as a 1 socket, 4 core, 2 thread CPU.
IOW we haven't really got 8 CPUs, more like equivalent of 5 CPUs.
as HT only really gives a x1.3 boost in best case, and I suspect
builds are not likely to be hitting the best case.
Can we afford to add 50% more load on the machine without making it
unusable? I don't know. But I think it would be worthwhile to at
least try and see how it handles an additional 25%, which is exactly
what this series does.
Giving it a try is ok I guess.
I expect there's probably more we can do to optimize the setup
too.
For example, what actual features of qcow2 are we using ? We're
not snapshotting VMs, we don't need grow-on-demand allocation.
AFACT we're paying the performance cost of qcow2 (l1/l2 table
lookups & metadata caching), for no reason. Switch the VMs to
fully pre-allocated raw files may improve I/O performance.
Raw LVM VGs would be even better but that will be painful
to setup given the host install setup.
I also wonder if we have the optimal aio setting for disks,
as there's nothing in the XML.
We could consider using cache=unsafe for VMs, though for
that I think we'd want to separate off a separate disk
for /home/jenkins so that if there was a host OS crash,
we wouldn't have to rebuild the entire VMs - just throw
away the data disk & recreate.
Since we've got plenty of RAM, another obvious thing would be
to turn on huge pages and use them for all guest RAM. This may
well have a very significant performance boost from reducing
CPU overhead which is our biggest bottleneck.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|