The dashboard itself simply takes away focus from everything else that makes sense to have in the CI article, so move it to it's own article and link it from the main CI article. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci-dashboard.rst | 216 ++++++++++++++++++++++++++++++++++++++++++ docs/ci.rst | 192 +------------------------------------ docs/meson.build | 1 + 3 files changed, 219 insertions(+), 190 deletions(-) create mode 100644 docs/ci-dashboard.rst diff --git a/docs/ci-dashboard.rst b/docs/ci-dashboard.rst new file mode 100644 index 0000000000..a7f4e71f96 --- /dev/null +++ b/docs/ci-dashboard.rst @@ -0,0 +1,216 @@ +=================== +GitLab CI Dashboard +=================== + +The dashboard below shows the current status of the GitLab CI jobs for each +repository: + +Core project +------------ + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt + - .. image:: https://gitlab.com/libvirt/libvirt/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt/pipelines + :alt: libvirt pipeline status + + +Language bindings +----------------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-csharp + - .. image:: https://gitlab.com/libvirt/libvirt-csharp/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-csharp/pipelines + :alt: libvirt-csharp pipeline status + + * - libvirt-go-module + - .. image:: https://gitlab.com/libvirt/libvirt-go-module/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-go-module/pipelines + :alt: libvirt-go-module pipeline status + + * - libvirt-java + - .. image:: https://gitlab.com/libvirt/libvirt-java/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-java/pipelines + :alt: libvirt-java pipeline status + + * - libvirt-ocaml + - .. image:: https://gitlab.com/libvirt/libvirt-ocaml/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-ocaml/pipelines + :alt: libvirt-ocaml pipeline status + + * - libvirt-perl + - .. image:: https://gitlab.com/libvirt/libvirt-perl/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-perl/pipelines + :alt: libvirt-perl pipeline status + + * - libvirt-php + - .. image:: https://gitlab.com/libvirt/libvirt-php/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-php/pipelines + :alt: libvirt-php pipeline status + + * - libvirt-python + - .. image:: https://gitlab.com/libvirt/libvirt-python/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-python/pipelines + :alt: libvirt-python pipeline status + + * - libvirt-ruby + - .. image:: https://gitlab.com/libvirt/libvirt-ruby/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-ruby/pipelines + :alt: libvirt-ruby pipeline status + + * - libvirt-rust + - .. image:: https://gitlab.com/libvirt/libvirt-rust/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-rust/pipelines + :alt: libvirt-rust pipeline status + + +Object mappings +--------------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-cim + - .. image:: https://gitlab.com/libvirt/libvirt-cim/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-cim/pipelines + :alt: libvirt-cim pipeline status + + * - libvirt-dbus + - .. image:: https://gitlab.com/libvirt/libvirt-dbus/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-dbus/pipelines + :alt: libvirt-dbus pipeline status + + * - libvirt-glib + - .. image:: https://gitlab.com/libvirt/libvirt-glib/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-glib/pipelines + :alt: libvirt-glib pipeline status + + * - libvirt-go-xml-module + - .. image:: https://gitlab.com/libvirt/libvirt-go-xml-module/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-go-xml-module/pipelines + :alt: libvirt-go-xml-module pipeline status + + * - libvirt-snmp + - .. image:: https://gitlab.com/libvirt/libvirt-snmp/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-snmp/pipelines + :alt: libvirt-snmp pipeline status + + +Testing +------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-ci + - .. image:: https://gitlab.com/libvirt/libvirt-ci/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-ci/pipelines + :alt: libvirt-ci pipeline status + + * - libvirt-test-API + - .. image:: https://gitlab.com/libvirt/libvirt-test-API/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-test-API/pipelines + :alt: libvirt-test-API pipeline status + + * - libvirt-tck + - .. image:: https://gitlab.com/libvirt/libvirt-tck/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-tck/pipelines + :alt: libvirt-tck pipeline status + + +Documentation / websites +------------------------ + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-publican + - .. image:: https://gitlab.com/libvirt/libvirt-publican/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-publican/pipelines + :alt: libvirt-publican pipeline status + + * - libvirt-appdev-guide-python + - .. image:: https://gitlab.com/libvirt/libvirt-appdev-guide-python/badges/master/pipelin... + :target: https://gitlab.com/libvirt/libvirt-appdev-guide-python/pipelines + :alt: libvirt-appdev-guide-python pipeline status + + * - libvirt-wiki + - .. image:: https://gitlab.com/libvirt/libvirt-wiki/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-wiki/pipelines + :alt: libvirt-wiki pipeline status + + * - virttools-planet + - .. image:: https://gitlab.com/libvirt/virttools-planet/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/virttools-planet/pipelines + :alt: virttools-planet pipeline status + + * - virttools-web + - .. image:: https://gitlab.com/libvirt/virttools-web/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/virttools-web/pipelines + :alt: virttools-web pipeline status + + +Miscellaneous +------------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-console-proxy + - .. image:: https://gitlab.com/libvirt/libvirt-console-proxy/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-console-proxy/pipelines + :alt: libvirt-console-proxy pipeline status + + * - libvirt-designer + - .. image:: https://gitlab.com/libvirt/libvirt-designer/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-designer/pipelines + :alt: libvirt-designer pipeline status + + * - libvirt-devaddr + - .. image:: https://gitlab.com/libvirt/libvirt-devaddr/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-devaddr/pipelines + :alt: libvirt-devaddr pipeline status + + * - libvirt-sandbox + - .. image:: https://gitlab.com/libvirt/libvirt-sandbox/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-sandbox/pipelines + :alt: libvirt-sandbox pipeline status + + * - libvirt-sandbox-image + - .. image:: https://gitlab.com/libvirt/libvirt-sandbox-image/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-sandbox-image/pipelines + :alt: libvirt-sandbox-image pipeline status + + * - libvirt-security-notice + - .. image:: https://gitlab.com/libvirt/libvirt-security-notice/badges/master/pipeline.sv... + :target: https://gitlab.com/libvirt/libvirt-security-notice/pipelines + :alt: libvirt-security-notice pipeline status diff --git a/docs/ci.rst b/docs/ci.rst index 71418d74bf..d5282c4d25 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -5,6 +5,8 @@ Libvirt Continuous Integration .. contents:: The libvirt project uses GitLab CI for automated testing. +`Here's <ci-dashboard.html>`__ our CI dashboard which shows the current status +of our pipelines. Linux builds and cross-compiled Windows builds happen on GitLab CI's shared runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI @@ -13,217 +15,27 @@ runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI Most of the tooling used to build CI pipelines is maintained as part of the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci>`_ subproject. -GitLab CI Dashboard -=================== -The dashboard below shows the current status of the GitLab CI jobs for each -repository: -Core project ------------- -.. list-table:: - :widths: 80 20 - :header-rows: 1 - * - Project - - Pipeline - * - libvirt - - .. image:: https://gitlab.com/libvirt/libvirt/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt/pipelines - :alt: libvirt pipeline status -Language bindings ------------------ -.. list-table:: - :widths: 80 20 - :header-rows: 1 - * - Project - - Pipeline - * - libvirt-csharp - - .. image:: https://gitlab.com/libvirt/libvirt-csharp/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-csharp/pipelines - :alt: libvirt-csharp pipeline status - * - libvirt-go-module - - .. image:: https://gitlab.com/libvirt/libvirt-go-module/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-go-module/pipelines - :alt: libvirt-go-module pipeline status - * - libvirt-java - - .. image:: https://gitlab.com/libvirt/libvirt-java/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-java/pipelines - :alt: libvirt-java pipeline status - * - libvirt-ocaml - - .. image:: https://gitlab.com/libvirt/libvirt-ocaml/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-ocaml/pipelines - :alt: libvirt-ocaml pipeline status - * - libvirt-perl - - .. image:: https://gitlab.com/libvirt/libvirt-perl/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-perl/pipelines - :alt: libvirt-perl pipeline status - * - libvirt-php - - .. image:: https://gitlab.com/libvirt/libvirt-php/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-php/pipelines - :alt: libvirt-php pipeline status - * - libvirt-python - - .. image:: https://gitlab.com/libvirt/libvirt-python/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-python/pipelines - :alt: libvirt-python pipeline status - * - libvirt-ruby - - .. image:: https://gitlab.com/libvirt/libvirt-ruby/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-ruby/pipelines - :alt: libvirt-ruby pipeline status - - * - libvirt-rust - - .. image:: https://gitlab.com/libvirt/libvirt-rust/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-rust/pipelines - :alt: libvirt-rust pipeline status -Object mappings ---------------- -.. list-table:: - :widths: 80 20 - :header-rows: 1 - * - Project - - Pipeline - * - libvirt-cim - - .. image:: https://gitlab.com/libvirt/libvirt-cim/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-cim/pipelines - :alt: libvirt-cim pipeline status - * - libvirt-dbus - - .. image:: https://gitlab.com/libvirt/libvirt-dbus/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-dbus/pipelines - :alt: libvirt-dbus pipeline status - * - libvirt-glib - - .. image:: https://gitlab.com/libvirt/libvirt-glib/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-glib/pipelines - :alt: libvirt-glib pipeline status - - * - libvirt-go-xml-module - - .. image:: https://gitlab.com/libvirt/libvirt-go-xml-module/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-go-xml-module/pipelines - :alt: libvirt-go-xml-module pipeline status - - * - libvirt-snmp - - .. image:: https://gitlab.com/libvirt/libvirt-snmp/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-snmp/pipelines - :alt: libvirt-snmp pipeline status - - -Testing -------- - -.. list-table:: - :widths: 80 20 - :header-rows: 1 - - * - Project - - Pipeline - - * - libvirt-ci - - .. image:: https://gitlab.com/libvirt/libvirt-ci/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-ci/pipelines - :alt: libvirt-ci pipeline status - - * - libvirt-test-API - - .. image:: https://gitlab.com/libvirt/libvirt-test-API/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-test-API/pipelines - :alt: libvirt-test-API pipeline status - - * - libvirt-tck - - .. image:: https://gitlab.com/libvirt/libvirt-tck/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-tck/pipelines - :alt: libvirt-tck pipeline status - - -Documentation / websites ------------------------- - -.. list-table:: - :widths: 80 20 - :header-rows: 1 - - * - Project - - Pipeline - * - libvirt-publican - - .. image:: https://gitlab.com/libvirt/libvirt-publican/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-publican/pipelines - :alt: libvirt-publican pipeline status - - * - libvirt-appdev-guide-python - - .. image:: https://gitlab.com/libvirt/libvirt-appdev-guide-python/badges/master/pipelin... - :target: https://gitlab.com/libvirt/libvirt-appdev-guide-python/pipelines - :alt: libvirt-appdev-guide-python pipeline status - - * - libvirt-wiki - - .. image:: https://gitlab.com/libvirt/libvirt-wiki/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-wiki/pipelines - :alt: libvirt-wiki pipeline status - - * - virttools-planet - - .. image:: https://gitlab.com/libvirt/virttools-planet/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/virttools-planet/pipelines - :alt: virttools-planet pipeline status - - * - virttools-web - - .. image:: https://gitlab.com/libvirt/virttools-web/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/virttools-web/pipelines - :alt: virttools-web pipeline status - - -Miscellaneous -------------- - -.. list-table:: - :widths: 80 20 - :header-rows: 1 - - * - Project - - Pipeline - - * - libvirt-console-proxy - - .. image:: https://gitlab.com/libvirt/libvirt-console-proxy/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-console-proxy/pipelines - :alt: libvirt-console-proxy pipeline status - - * - libvirt-designer - - .. image:: https://gitlab.com/libvirt/libvirt-designer/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-designer/pipelines - :alt: libvirt-designer pipeline status - - * - libvirt-devaddr - - .. image:: https://gitlab.com/libvirt/libvirt-devaddr/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-devaddr/pipelines - :alt: libvirt-devaddr pipeline status - - * - libvirt-sandbox - - .. image:: https://gitlab.com/libvirt/libvirt-sandbox/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-sandbox/pipelines - :alt: libvirt-sandbox pipeline status - - * - libvirt-sandbox-image - - .. image:: https://gitlab.com/libvirt/libvirt-sandbox-image/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-sandbox-image/pipelines - :alt: libvirt-sandbox-image pipeline status - - * - libvirt-security-notice - - .. image:: https://gitlab.com/libvirt/libvirt-security-notice/badges/master/pipeline.sv... - :target: https://gitlab.com/libvirt/libvirt-security-notice/pipelines - :alt: libvirt-security-notice pipeline status diff --git a/docs/meson.build b/docs/meson.build index cb70ef6084..fd9df047cb 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -34,6 +34,7 @@ docs_rst_files = [ 'bugs', 'cgroups', 'ci', + 'ci-dashboard', 'coding-style', 'committer-guidelines', 'compiling', -- 2.36.1

Most importantly, how to get it, how install dependencies and how to run it. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/testtck.rst | 92 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 11 deletions(-) diff --git a/docs/testtck.rst b/docs/testtck.rst index 51de095657..19f1cc5726 100644 --- a/docs/testtck.rst +++ b/docs/testtck.rst @@ -2,6 +2,8 @@ libvirt TCK : Technology Compatibility Kit ========================================== +.. contents:: + The libvirt TCK provides a framework for performing testing of the integration between libvirt drivers, the underlying virt hypervisor technology, related operating system services and system configuration. The idea (and name) is @@ -23,15 +25,83 @@ determine the level of compatibility of their platform, and evaluate whether it will meet their needs, and get awareness of any regressions that may have occurred since a previous test run. -For more details you can look at: - -- The initial `mail from Daniel - Berrange <https://www.redhat.com/archives/libvir-list/2009-April/msg00176.html>`__ - presenting the project. -- The `page describing - VirtTCK <https://fedoraproject.org/wiki/Features/VirtTCK>`__ the inclusion of - libvirt-TCK as a Fedora Feature. - Libvirt-TCK is maintained using `a GIT -repository <https://gitlab.com/libvirt/libvirt-tck>`__, and comment, patches and -reviews are carried on the `libvir-list <contact.html>`__ development list. +repository <https://gitlab.com/libvirt/libvirt-tck>`__. GitLab is also the place +where the whole TCK development workflow (issues, merge requests, comments) +happens. + +Using TCK +--------- + +TCK can be used independently of the enviroment, i.e. both on your local host +or in a VM. We strongly recommend using a VM for the tests as TCK might affect +your current host setup, see `Running TCK`_. + +Installing dependencies +~~~~~~~~~~~~~~~~~~~~~~~ + +Since TCK is based on libvirt Perl bindings, you'll need to have the proper +version of the bindings installed for the version of libvirt you wish to test +in order to be able execute the TCK test suite successfully. Additionally, a +number of Perl dependencies will need to be installed as well, some will be +available through the system package manager and some will likely need to be +installed from CPAN (Perl's equivalent of Python's PyPI). Here's where +`libvirt-ci's <https://gitlab.com/libvirt/libvirt-ci.git>`__ lcitool can help +with preparing a test environment in a fresh VM, taking care of the +dependencies along the way: + +:: + + $ lcitool install --target fedora-36 tck-fedora36 --wait + +would get you a new Fedora 36 VM named ``tck-fedora36``. Then + +:: + + $ lcitool update tck-fedora36 libvirt,libvirt-perl,libvirt-tck+runtime + +will install all the necessary dependencies to build libvirt, the corresponding +Perl bindings and all TCK runtime dependencies to be able to execute the tests. +We also recommend executing TCK using the Avocado framework as the test harness +engine which means that you'll have to install Avocado in the test environment +as well. You can get it either from +`PyPI <https://pypi.org/project/avocado-framework/>`__ (recommended), or if +you're on Fedora you can make use of the Avocado `module <https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html#installing-from-packages>`__. +Using Avocado is not mandatory for the time being and you can skip it, but +in the future we plan on making the TCK internal coupling with Avocado tighter. + +Running TCK +~~~~~~~~~~~ + +Once you have all the dependencies installed, you can then proceed with running +as root the test suite as root (when running with Avocado): + +:: + + # avocado --config avocado.config run --tap - ./scripts/ + +from the TCK's git root. + + +If you don't want to install Avocado you can execute tests using the +``libvirt-tck`` binary directly (again, from the git root). You'll need to pass +a few options that Avocado takes care of: + +:: + + # PERL5LIB=./lib perl bin/libvirt-tck -c <path_to_config> --force ./scripts + +Note that running with root privileges is necessary since some tests need +access to system resources or configs. This, along with the fact that some +tests might affect the host system are good reasons to consider using a test VM +as described above. + +Contributing a test +------------------- + +We'd appreciate if you provided a functional test case whenever you're adding a +new feature or fixing a bug in libvirt with the only complication being that +in case you're adding a new public API then a Perl binding will have to be +introduced first. After that, the best way to start is looking at some existing +tests, copy-pasting one that fits your scenario the best and tweak the +remaining bits. -- 2.36.1

Since running our functional test suite in GitLab cannot make use of the shared resources it makes sense to document the process of adding own HW to run the custom libvirt executor that powers the integration suite. This article will likely make even more sense in the future with GitLab severely cutting down on shared CI resources. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci-runners.rst | 86 +++++++++++++++++++++++++++++++++++++++++++++ docs/meson.build | 1 + 2 files changed, 87 insertions(+) create mode 100644 docs/ci-runners.rst diff --git a/docs/ci-runners.rst b/docs/ci-runners.rst new file mode 100644 index 0000000000..fd5fdd121a --- /dev/null +++ b/docs/ci-runners.rst @@ -0,0 +1,86 @@ +GitLab CI Custom (Specific) Runners +=================================== + +.. contents:: + +GitLab's CI allows additional machines to be added to the project's or group's +pool of runners (a runner is a machine running the GitLab's +`gitlab-runner <https://gitlab.com/gitlab-org/gitlab-runner/>`__ agent service). +Upon registering the runner the runner will then be ready accepting CI jobs +depending on the pipeline configuration. Unlike the shared runners provided +directly by GitLab's hosted SaaS specific runners are only used within the +project/group which they were registered to, so you don't need to worry about +forks burning CPU cycles on your precious HW resources. + +Understandably, we respect your decision to keep your runners only visible to +your fork, but for the sake of the community we'd appreciate if you decided to +register your runner with the upstream libvirt project instead. As we're only +interested in running upstream test workloads (which you can even help +defining) maintenance and security of the HW is your own responsibility and so +we can promise to never ask for physical or remote access to the machine. + +Machine Setup Howto +------------------- + +The following sections will guide you through the necessary setup of your +specific GitLab runners. + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that is supposed +to run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears-like icon at the bottom of the left hand side + vertical toolbar), then + * CI/CD, then + * Runners, and click on the *Expand* button, then + * Under *Set up a specific Runner manually*, look for the value under + *And this registration token:* + +Note that in order to register a runner with the upstream libvirt project +you'll need to work with the project maintainers to successfully register your +machine. + +Following the `registration <https://docs.gitlab.com/runner/register/>`__ +process, it's necessary to configure the runner tags, and optionally other +configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the *Expand* button, then + * *Runners activated for this project*, then + * Click on the *Edit* icon (next to the *Lock* Icon) + +Don't forget to add a tag to your runner as these are used to route specific +jobs to specific runners, e.g. if a job in ``ci/integration.yml`` looked like +this :: + + centos-stream-9-tests: + ... + variables: + # needed by libvirt-gitlab-executor + DISTRO: centos-stream-9 + # can be overridden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: my-vm-host + tags: + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG + +it would mean that the CentOS Stream 9 job would only be scheduled on runners +bearing the 'my-vm-host' tag. + +Running integration tests +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Libvirt's integration tests run in a nested virtualization environment. So, if +you wish to run integration tests on your bare-metal machine, you'll have to +make use of GitLab's +`custom executor <https://docs.gitlab.com/runner/executors/custom.html>`__ +feature which allows you to provision any kind of environment for a workload to +run - in libvirt's case - a virtual machine. If you need any help with creating +VM template images ready to run libvirt's integration test suite, have a look +at the `libvirt-gitlab-executor <https://gitlab.com/libvirt/libvirt-custom-executor>`__ +project which encapsulates provisioning, execution, and teardown of the +virtualized environments in a single tool. diff --git a/docs/meson.build b/docs/meson.build index fd9df047cb..0c7c54a75f 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -35,6 +35,7 @@ docs_rst_files = [ 'cgroups', 'ci', 'ci-dashboard', + 'ci-runners', 'coding-style', 'committer-guidelines', 'compiling', -- 2.36.1

Currently we don't have much information on how testing is done in libvirt and the little we have is scattered among multiple files. This patch creates a common landing page containing all important bits about testing in libvirt, providing links to respective sections which deserve their own articles. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/meson.build | 1 + docs/testing.rst | 176 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 177 insertions(+) create mode 100644 docs/testing.rst diff --git a/docs/meson.build b/docs/meson.build index 0c7c54a75f..6d61e18385 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -105,6 +105,7 @@ docs_rst_files = [ 'submitting-patches', 'support', 'testapi', + 'testing', 'testsuites', 'testtck', 'uri', diff --git a/docs/testing.rst b/docs/testing.rst new file mode 100644 index 0000000000..af0f334633 --- /dev/null +++ b/docs/testing.rst @@ -0,0 +1,176 @@ +======= +Testing +======= + +.. contents:: + +Different types of tests are available to libvirt developers for testing a +given libvirt release. + +Unit tests +---------- + +The unit test suite present in the source code is mainly used to test our +XML parser/formatter, QEMU command line generator, QEMU capabilities probing, +etc. It is run by developers before submitting patches upstream and is +mandatory to pass for any contribution to be accepted upstream. One can run +the test suite in the source tree with the following + +:: + + make check (libvirt 6.6.0 and older) + +:: + + ninja test (libvirt 6.7.0 and newer) + + +Container builds +---------------- + +Technically speaking these are not tests in the common sense. However, usage of +public container images to build libvirt in predefined and widely accessible +environments makes it possible to expand our coverage across distros, +architectures, toolchain flavors and library versions and as such is a very +valuable marker when accepting upstream contributions. Therefore, it is +recommended to run libvirt builds against your changes in various containers to +either locally or by using GitLab's shared CI runners to make sure everything +runs cleanly before submitting your patches. The images themselves come from +libvirt's GitLab container registry, but this can be overriden if needed, see +below. + +Registry +~~~~~~~~ + +Libvirt project has its container registry hosted by GitLab at +``registry.gitlab.com/libvirt/libvirt`` which will automatically be +used to pull in pre-built layers. This avoids the need to build all the +containers locally using the Dockerfile recipes found in ``ci/containers/``. + + +Running container builds with GitLab CI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As long as your GitLab account has CI minutes available, pipelines will run +automatically on every branch push to your fork. + +Running container builds locally +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to run container builds locally, we have a ``helper`` script inside +the ``ci`` directory that can pull, build, and test (if applicable) changes on +your current local branch. It supports both the Docker and Podman runtimes +with an automatic selection of whichever runtime is configured on your system. +In case neither has been enabled/configured, please go through the following +prerequisites. + +Docker Prerequisites +~~~~~~~~~~~~~~~~~~~~ + +Install "docker" with the system package manager and start the Docker service +on your development machine, then make sure you have the privilege to run +Docker commands. Typically it means setting up passwordless ``sudo docker`` +command or login as root. For example: + +.. code:: + + $ sudo yum install docker + $ # or `apt-get install docker` for Ubuntu, etc. + $ sudo systemctl start docker + $ sudo docker ps + +The last command should print an empty table, to verify the system is ready. + +An alternative method to set up permissions is by adding the current user to +"docker" group and making the docker daemon socket file (by default +``/var/run/docker.sock``) accessible to the group: + +.. code:: + + $ sudo groupadd docker + $ sudo usermod $USER -a -G docker + $ sudo chown :docker /var/run/docker.sock + +Note that any one of above configurations makes it possible for the user to +exploit the whole host with Docker bind mounting or other privileged +operations. So only do it on development machines. + +Podman Prerequisites +~~~~~~~~~~~~~~~~~~~~ + +Install "podman" with the system package manager. + +.. code:: + + $ sudo dnf install podman + $ podman ps + +The last command should print an empty table, to verify the system is ready. + +Examples of executing local container builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All of the following examples will utilize ``helper`` script mentioned earlier +sections. Let's start with the basics - listing available container images in +the default libvirt registry: + +:: + + $ cd <libvirt_git>/ci + $ ./helper --help + $ ./helper list-images + Available x86 container images: + + ... + alpine-edge + fedora-rawhide + ... + + Available cross-compiler container images: + + ... + debian-sid-cross-s390x + fedora-rawhide-cross-mingw32 + fedora-rawhide-cross-mingw64 + ... + +Now let's say one would want to build their local libvirt changes on Alpine +Edge using their own GitLab's registry container. They'd then proceed with + +:: + + $ ci/helper build --image-prefix registry.gitlab.com/<user>/libvirt/ci- alpine-edge + +Finally, it would be nice if one could get an interactive shell inside the +test environment to debug potential build issues. This can be achieved with the +following: + +:: + + $ ci/helper shell alpine-edge + + +Integration tests +----------------- + +There are a few frameworks for writing and running functional tests in libvirt +with TCK being the one that runs in our upstream CI. + +- the `TCK test suite <testtck.html>`__ is a functional test suite implemented + using the `Perl bindings <https://search.cpan.org/dist/Sys-Virt/>`__ of + libvirt. This is the recommended framework to use for writing upstream + functional tests at the moment. You can start by cloning the + `TCK git repo <https://gitlab.com/libvirt/libvirt-tck>`__. + +- the `Avocado VT <https://github.com/avocado-framework/avocado-vt>`__ test + suite with the libvirt plugin is another framework implementing functional + testing utilizing the Avocado test framework underneath. Although written in + Python, the vast majority of the tests are exercising libvirt through the + command line client ``virsh``. + +- the `libvirt-test-API <testapi.html>`__ is also a functional test suite, but + implemented using the `Python bindings <python.html>`__ of libvirt. + Unfortunately this framework is the least recommended one as it's largely + unmaintained and may be completely deprecated in the future in favour of TCK. + You can get it by cloning the + `git repo <https://gitlab.com/libvirt/libvirt-test-API/>`__. -- 2.36.1

The new article provides more in-depth information on testing options in libvirt. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/docs.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/docs.rst b/docs/docs.rst index 9fd8b7c37e..b17d73743a 100644 --- a/docs/docs.rst +++ b/docs/docs.rst @@ -154,10 +154,8 @@ Project development `API extensions <api_extension.html>`__ Adding new public libvirt APIs -`Functional testing <testsuites.html>`__ - Testing libvirt with - `TCK test suite <testtck.html>`__ and - `Libvirt-test-API <testapi.html>`__ +`Testing <testing.html>`__ + Details various types of testing available for libvirt `New repo setup <newreposetup.html>`__ Procedure for configuring new git repositories for libvirt -- 2.36.1

The article was replaced with a new one in previous commit, so we don't need this one anymore. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/meson.build | 1 - docs/testsuites.rst | 37 ------------------------------------- 2 files changed, 38 deletions(-) delete mode 100644 docs/testsuites.rst diff --git a/docs/meson.build b/docs/meson.build index 6d61e18385..0b16d64488 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -106,7 +106,6 @@ docs_rst_files = [ 'support', 'testapi', 'testing', - 'testsuites', 'testtck', 'uri', 'windows', diff --git a/docs/testsuites.rst b/docs/testsuites.rst deleted file mode 100644 index 4fc733e615..0000000000 --- a/docs/testsuites.rst +++ /dev/null @@ -1,37 +0,0 @@ -=========== -Test suites -=========== - -There is a few test suites available to developers for testing a given version -of libvirt: - -- the internal test suite: present in the source code, it is run by developers - before submitting patches upstream, it is also suggested to have it run and - pass as part of the packaging process for distributions. It is run by - launching: - - :: - - make check (libvirt 6.6.0 and older) - - :: - - ninja test (libvirt 6.7.0 and newer) - - in a source tree after compilation has finished. It doesn't really make - functional testing but checks that large portions of the code not interacting - directly with virtualization functions properly. - -- the `TCK test suite <testtck.html>`__ is a functional test suite implemented - using the `Perl bindings <https://search.cpan.org/dist/Sys-Virt/>`__ of - libvirt. It is available separately as a - `download <ftp://libvirt.org/libvirt/tck/>`__, as a - `package <https://rpmfind.net/linux/rpm2html/search.php?query=libvirt-tck>`__ - in Fedora distributions, but best is probably to get the `version from - GIT <https://gitlab.com/libvirt/libvirt-tck>`__. - -- the `libvirt-test-API <testapi.html>`__ is also a functional test suite, but - implemented using the `Python bindings <python.html>`__ of libvirt. It is - available separately as a - `download <ftp://libvirt.org/libvirt/libvirt-test-API/>`__, or directly get - the `version from GIT <https://gitlab.com/libvirt/libvirt-test-API/>`__. -- 2.36.1

It's not just strategy the master CI article talks (or will talk in the future) about. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/docs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs.rst b/docs/docs.rst index b17d73743a..8ca0afdc1a 100644 --- a/docs/docs.rst +++ b/docs/docs.rst @@ -136,8 +136,8 @@ Project development `Project strategy <strategy.html>`__ Sets a vision for future direction & technical choices -`CI Testing <ci.html>`__ - Details of the Continuous Integration testing strategy +`CI <ci.html>`__ + Details on our Continuous Integration `Bug reports <bugs.html>`__ How and where to report bugs and request features -- 2.36.1

Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/ci.rst b/docs/ci.rst index d5282c4d25..bef023c521 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -8,6 +8,9 @@ The libvirt project uses GitLab CI for automated testing. `Here's <ci-dashboard.html>`__ our CI dashboard which shows the current status of our pipelines. +Libvirt builds +============== + Linux builds and cross-compiled Windows builds happen on GitLab CI's shared runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI <https://cirrus-ci.com/>`_ jobs behind the scenes. @@ -15,11 +18,28 @@ runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI Most of the tooling used to build CI pipelines is maintained as part of the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci>`_ subproject. +Integration tests +================= +Integration tests in our CI pipelines require dedicated HW which is not +available to forks, see `GitLab CI Custom Runners <ci-runners.html>`__. +Therefore, in order to execute the integration tests as part of your libvirt +fork's GitLab CI you'll need to provide your own runner. You'll also need to +set a few CI variables to run the integration tests as part of the CI pipeline, +see below. +GitLab CI variables +------------------- +* ``LIBVIRT_CI_INTEGRATION`` - enables integration test runs manually or in forks +* ``LIBVIRT_CI_INTEGRATION_RUNNER_TAG`` - overrides the upstream runner tag on the +Retrieving test logs +-------------------- +In case the integration test suite fails in our CI pipelines, a job artifact is +generated containing Avocado logs, libvirt debug logs, and the latest traceback +(if one was produced during a daemon's execution). -- 2.36.1

Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/ci.rst b/docs/ci.rst index bef023c521..ea1b839eac 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -41,21 +41,60 @@ In case the integration test suite fails in our CI pipelines, a job artifact is generated containing Avocado logs, libvirt debug logs, and the latest traceback (if one was produced during a daemon's execution). +Adding new OS platforms OR build pre-requisites +=============================================== +Since all of the Dockerfiles libvirt uses for CI have been generated by ``lcitool`` +provided by the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci.git>`__ project, +most relevant changes will need to be introduced to ``lcitool`` first. +In cases where ``libvirt-ci`` does not know about the OS distro/build +pre-requisite that is desired to be tested, proceed with the following: + * Send a mail regarding your desired build/coverage change to libvir-list + Note that in case you're looking into adding a new OS platform there are + limited CI compute resources available to libvirt, so the cost/benefit + trade-off of adding new OS distros needs to be considered and + is subject to a discussion on the mailing list. + * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues + pointing to the libvir-list mail thread in the archives. + This alerts other people who might be interested in the work + to avoid duplication, as well as to get feedback from libvirt-ci + maintainers on any tips to ease the addition. +Assuming there is agreement to the change you wish to make then + #. Fork the ``libvirt-ci`` project on gitlab + #. + * If adding a new OS distro, add metadata under + ``guests/lcitool/lcitool/ansible/group_vars/`` + for the new OS distro. There might be code changes required if + the OS distro uses a package format not currently known. The + ``libvirt-ci`` maintainers can advise on this when the issue + is file. Edit the ``mappings.yml`` file to update all the existing package + entries, providing details of the new OS distro **OR** + * If **NOT** adding a new OS distro, simply edit the ``mappings.yml`` file + to provide a distro-agnostic mapping for the pre-requisite you're adding + #. Commit the ``mappings.yml`` change and submit a merge request to + the ``libvirt-ci`` project + #. CI pipeline will run to validate that the changes to ``mappings.yml`` + are correct, by attempting to install the newly listed package on + all OS distributions supported by ``libvirt-ci`` + #. Once the merge request is accepted, go back to libvirt and update the + ``ci/manifest.yml`` file to reflect your change + #. From the root of the libvirt git repository run +:: + $ lcitool manifest -- 2.36.1

This is just a glue to the testing article introduced in previous commits. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/ci.rst b/docs/ci.rst index ea1b839eac..515a02a6f6 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -97,4 +97,8 @@ Assuming there is agreement to the change you wish to make then $ lcitool manifest +Running CI workloads locally +============================ +If you're interested in running the CI test workloads locally, please read +our `testing <testing.html>`__ guide. -- 2.36.1