[libvirt PATCH v2 00/10] docs: Expand documentation on the CI/testing topic
This series was inspired by QEMU's documentation on CI and focuses on the following: - adds documentation on how to add a custom runner to the libvirt project or its fork, mainly in context of running functional testing - replaces the old 'testsuites' article with a more detailed landing page describing different types of testing available in libvirt - updates the TCK documentation with details on how TCK should be installed and used nowadays Since v1: - described the --force argument with TCK usage - moved the article about adding new platforms to libvirt-ci and linked it from our docs -> corresponding MR here: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/295 -> ^the MR must merged **FIRST**! - moved Podman prerequistes for local container builds before Docker Here's a fresh website pipeline with artifacts: https://gitlab.com/eskultety/libvirt/-/jobs/2722840726 Erik Skultety (10): docs: Move the CI dashboard to its own RST module docs: Provide more information about the TCK test suite docs: Provide an article on how to add a custom runner to the project docs: Provide an article on testing docs: Replace testsuites article with the new 'testing' article docs: Drop the testsuites article docs: Change the CI headline from "CI Testing" to "CI" docs: ci: Add info about the two major types of jobs our CI runs docs: ci: Add a section on how to add a new platform to libvirt CI docs: ci: Add a brief section on how to run the CI workload locally docs/ci-dashboard.rst | 216 ++++++++++++++++++++++++++++++++++++++ docs/ci-runners.rst | 86 +++++++++++++++ docs/ci.rst | 237 ++++++------------------------------------ docs/docs.rst | 10 +- docs/meson.build | 4 +- docs/testing.rst | 172 ++++++++++++++++++++++++++++++ docs/testsuites.rst | 37 ------- docs/testtck.rst | 99 ++++++++++++++++-- 8 files changed, 602 insertions(+), 259 deletions(-) create mode 100644 docs/ci-dashboard.rst create mode 100644 docs/ci-runners.rst create mode 100644 docs/testing.rst delete mode 100644 docs/testsuites.rst -- 2.36.1
The dashboard itself simply takes away focus from everything else that makes sense to have in the CI article, so move it to it's own article and link it from the main CI article. Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/ci-dashboard.rst | 216 ++++++++++++++++++++++++++++++++++++++++++ docs/ci.rst | 192 +------------------------------------ docs/meson.build | 1 + 3 files changed, 219 insertions(+), 190 deletions(-) create mode 100644 docs/ci-dashboard.rst diff --git a/docs/ci-dashboard.rst b/docs/ci-dashboard.rst new file mode 100644 index 0000000000..a7f4e71f96 --- /dev/null +++ b/docs/ci-dashboard.rst @@ -0,0 +1,216 @@ +=================== +GitLab CI Dashboard +=================== + +The dashboard below shows the current status of the GitLab CI jobs for each +repository: + +Core project +------------ + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt + - .. image:: https://gitlab.com/libvirt/libvirt/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt/pipelines + :alt: libvirt pipeline status + + +Language bindings +----------------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-csharp + - .. image:: https://gitlab.com/libvirt/libvirt-csharp/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-csharp/pipelines + :alt: libvirt-csharp pipeline status + + * - libvirt-go-module + - .. image:: https://gitlab.com/libvirt/libvirt-go-module/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-go-module/pipelines + :alt: libvirt-go-module pipeline status + + * - libvirt-java + - .. image:: https://gitlab.com/libvirt/libvirt-java/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-java/pipelines + :alt: libvirt-java pipeline status + + * - libvirt-ocaml + - .. image:: https://gitlab.com/libvirt/libvirt-ocaml/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-ocaml/pipelines + :alt: libvirt-ocaml pipeline status + + * - libvirt-perl + - .. image:: https://gitlab.com/libvirt/libvirt-perl/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-perl/pipelines + :alt: libvirt-perl pipeline status + + * - libvirt-php + - .. image:: https://gitlab.com/libvirt/libvirt-php/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-php/pipelines + :alt: libvirt-php pipeline status + + * - libvirt-python + - .. image:: https://gitlab.com/libvirt/libvirt-python/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-python/pipelines + :alt: libvirt-python pipeline status + + * - libvirt-ruby + - .. image:: https://gitlab.com/libvirt/libvirt-ruby/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-ruby/pipelines + :alt: libvirt-ruby pipeline status + + * - libvirt-rust + - .. image:: https://gitlab.com/libvirt/libvirt-rust/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-rust/pipelines + :alt: libvirt-rust pipeline status + + +Object mappings +--------------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-cim + - .. image:: https://gitlab.com/libvirt/libvirt-cim/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-cim/pipelines + :alt: libvirt-cim pipeline status + + * - libvirt-dbus + - .. image:: https://gitlab.com/libvirt/libvirt-dbus/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-dbus/pipelines + :alt: libvirt-dbus pipeline status + + * - libvirt-glib + - .. image:: https://gitlab.com/libvirt/libvirt-glib/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-glib/pipelines + :alt: libvirt-glib pipeline status + + * - libvirt-go-xml-module + - .. image:: https://gitlab.com/libvirt/libvirt-go-xml-module/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-go-xml-module/pipelines + :alt: libvirt-go-xml-module pipeline status + + * - libvirt-snmp + - .. image:: https://gitlab.com/libvirt/libvirt-snmp/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-snmp/pipelines + :alt: libvirt-snmp pipeline status + + +Testing +------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-ci + - .. image:: https://gitlab.com/libvirt/libvirt-ci/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-ci/pipelines + :alt: libvirt-ci pipeline status + + * - libvirt-test-API + - .. image:: https://gitlab.com/libvirt/libvirt-test-API/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-test-API/pipelines + :alt: libvirt-test-API pipeline status + + * - libvirt-tck + - .. image:: https://gitlab.com/libvirt/libvirt-tck/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-tck/pipelines + :alt: libvirt-tck pipeline status + + +Documentation / websites +------------------------ + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-publican + - .. image:: https://gitlab.com/libvirt/libvirt-publican/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-publican/pipelines + :alt: libvirt-publican pipeline status + + * - libvirt-appdev-guide-python + - .. image:: https://gitlab.com/libvirt/libvirt-appdev-guide-python/badges/master/pipelin... + :target: https://gitlab.com/libvirt/libvirt-appdev-guide-python/pipelines + :alt: libvirt-appdev-guide-python pipeline status + + * - libvirt-wiki + - .. image:: https://gitlab.com/libvirt/libvirt-wiki/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-wiki/pipelines + :alt: libvirt-wiki pipeline status + + * - virttools-planet + - .. image:: https://gitlab.com/libvirt/virttools-planet/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/virttools-planet/pipelines + :alt: virttools-planet pipeline status + + * - virttools-web + - .. image:: https://gitlab.com/libvirt/virttools-web/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/virttools-web/pipelines + :alt: virttools-web pipeline status + + +Miscellaneous +------------- + +.. list-table:: + :widths: 80 20 + :header-rows: 1 + + * - Project + - Pipeline + + * - libvirt-console-proxy + - .. image:: https://gitlab.com/libvirt/libvirt-console-proxy/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-console-proxy/pipelines + :alt: libvirt-console-proxy pipeline status + + * - libvirt-designer + - .. image:: https://gitlab.com/libvirt/libvirt-designer/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-designer/pipelines + :alt: libvirt-designer pipeline status + + * - libvirt-devaddr + - .. image:: https://gitlab.com/libvirt/libvirt-devaddr/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-devaddr/pipelines + :alt: libvirt-devaddr pipeline status + + * - libvirt-sandbox + - .. image:: https://gitlab.com/libvirt/libvirt-sandbox/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-sandbox/pipelines + :alt: libvirt-sandbox pipeline status + + * - libvirt-sandbox-image + - .. image:: https://gitlab.com/libvirt/libvirt-sandbox-image/badges/master/pipeline.svg + :target: https://gitlab.com/libvirt/libvirt-sandbox-image/pipelines + :alt: libvirt-sandbox-image pipeline status + + * - libvirt-security-notice + - .. image:: https://gitlab.com/libvirt/libvirt-security-notice/badges/master/pipeline.sv... + :target: https://gitlab.com/libvirt/libvirt-security-notice/pipelines + :alt: libvirt-security-notice pipeline status diff --git a/docs/ci.rst b/docs/ci.rst index 71418d74bf..d5282c4d25 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -5,6 +5,8 @@ Libvirt Continuous Integration .. contents:: The libvirt project uses GitLab CI for automated testing. +`Here's <ci-dashboard.html>`__ our CI dashboard which shows the current status +of our pipelines. Linux builds and cross-compiled Windows builds happen on GitLab CI's shared runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI @@ -13,217 +15,27 @@ runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI Most of the tooling used to build CI pipelines is maintained as part of the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci>`_ subproject. -GitLab CI Dashboard -=================== -The dashboard below shows the current status of the GitLab CI jobs for each -repository: -Core project ------------- -.. list-table:: - :widths: 80 20 - :header-rows: 1 - * - Project - - Pipeline - * - libvirt - - .. image:: https://gitlab.com/libvirt/libvirt/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt/pipelines - :alt: libvirt pipeline status -Language bindings ------------------ -.. list-table:: - :widths: 80 20 - :header-rows: 1 - * - Project - - Pipeline - * - libvirt-csharp - - .. image:: https://gitlab.com/libvirt/libvirt-csharp/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-csharp/pipelines - :alt: libvirt-csharp pipeline status - * - libvirt-go-module - - .. image:: https://gitlab.com/libvirt/libvirt-go-module/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-go-module/pipelines - :alt: libvirt-go-module pipeline status - * - libvirt-java - - .. image:: https://gitlab.com/libvirt/libvirt-java/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-java/pipelines - :alt: libvirt-java pipeline status - * - libvirt-ocaml - - .. image:: https://gitlab.com/libvirt/libvirt-ocaml/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-ocaml/pipelines - :alt: libvirt-ocaml pipeline status - * - libvirt-perl - - .. image:: https://gitlab.com/libvirt/libvirt-perl/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-perl/pipelines - :alt: libvirt-perl pipeline status - * - libvirt-php - - .. image:: https://gitlab.com/libvirt/libvirt-php/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-php/pipelines - :alt: libvirt-php pipeline status - * - libvirt-python - - .. image:: https://gitlab.com/libvirt/libvirt-python/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-python/pipelines - :alt: libvirt-python pipeline status - * - libvirt-ruby - - .. image:: https://gitlab.com/libvirt/libvirt-ruby/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-ruby/pipelines - :alt: libvirt-ruby pipeline status - - * - libvirt-rust - - .. image:: https://gitlab.com/libvirt/libvirt-rust/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-rust/pipelines - :alt: libvirt-rust pipeline status -Object mappings ---------------- -.. list-table:: - :widths: 80 20 - :header-rows: 1 - * - Project - - Pipeline - * - libvirt-cim - - .. image:: https://gitlab.com/libvirt/libvirt-cim/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-cim/pipelines - :alt: libvirt-cim pipeline status - * - libvirt-dbus - - .. image:: https://gitlab.com/libvirt/libvirt-dbus/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-dbus/pipelines - :alt: libvirt-dbus pipeline status - * - libvirt-glib - - .. image:: https://gitlab.com/libvirt/libvirt-glib/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-glib/pipelines - :alt: libvirt-glib pipeline status - - * - libvirt-go-xml-module - - .. image:: https://gitlab.com/libvirt/libvirt-go-xml-module/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-go-xml-module/pipelines - :alt: libvirt-go-xml-module pipeline status - - * - libvirt-snmp - - .. image:: https://gitlab.com/libvirt/libvirt-snmp/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-snmp/pipelines - :alt: libvirt-snmp pipeline status - - -Testing -------- - -.. list-table:: - :widths: 80 20 - :header-rows: 1 - - * - Project - - Pipeline - - * - libvirt-ci - - .. image:: https://gitlab.com/libvirt/libvirt-ci/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-ci/pipelines - :alt: libvirt-ci pipeline status - - * - libvirt-test-API - - .. image:: https://gitlab.com/libvirt/libvirt-test-API/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-test-API/pipelines - :alt: libvirt-test-API pipeline status - - * - libvirt-tck - - .. image:: https://gitlab.com/libvirt/libvirt-tck/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-tck/pipelines - :alt: libvirt-tck pipeline status - - -Documentation / websites ------------------------- - -.. list-table:: - :widths: 80 20 - :header-rows: 1 - - * - Project - - Pipeline - * - libvirt-publican - - .. image:: https://gitlab.com/libvirt/libvirt-publican/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-publican/pipelines - :alt: libvirt-publican pipeline status - - * - libvirt-appdev-guide-python - - .. image:: https://gitlab.com/libvirt/libvirt-appdev-guide-python/badges/master/pipelin... - :target: https://gitlab.com/libvirt/libvirt-appdev-guide-python/pipelines - :alt: libvirt-appdev-guide-python pipeline status - - * - libvirt-wiki - - .. image:: https://gitlab.com/libvirt/libvirt-wiki/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-wiki/pipelines - :alt: libvirt-wiki pipeline status - - * - virttools-planet - - .. image:: https://gitlab.com/libvirt/virttools-planet/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/virttools-planet/pipelines - :alt: virttools-planet pipeline status - - * - virttools-web - - .. image:: https://gitlab.com/libvirt/virttools-web/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/virttools-web/pipelines - :alt: virttools-web pipeline status - - -Miscellaneous -------------- - -.. list-table:: - :widths: 80 20 - :header-rows: 1 - - * - Project - - Pipeline - - * - libvirt-console-proxy - - .. image:: https://gitlab.com/libvirt/libvirt-console-proxy/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-console-proxy/pipelines - :alt: libvirt-console-proxy pipeline status - - * - libvirt-designer - - .. image:: https://gitlab.com/libvirt/libvirt-designer/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-designer/pipelines - :alt: libvirt-designer pipeline status - - * - libvirt-devaddr - - .. image:: https://gitlab.com/libvirt/libvirt-devaddr/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-devaddr/pipelines - :alt: libvirt-devaddr pipeline status - - * - libvirt-sandbox - - .. image:: https://gitlab.com/libvirt/libvirt-sandbox/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-sandbox/pipelines - :alt: libvirt-sandbox pipeline status - - * - libvirt-sandbox-image - - .. image:: https://gitlab.com/libvirt/libvirt-sandbox-image/badges/master/pipeline.svg - :target: https://gitlab.com/libvirt/libvirt-sandbox-image/pipelines - :alt: libvirt-sandbox-image pipeline status - - * - libvirt-security-notice - - .. image:: https://gitlab.com/libvirt/libvirt-security-notice/badges/master/pipeline.sv... - :target: https://gitlab.com/libvirt/libvirt-security-notice/pipelines - :alt: libvirt-security-notice pipeline status diff --git a/docs/meson.build b/docs/meson.build index cb70ef6084..fd9df047cb 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -34,6 +34,7 @@ docs_rst_files = [ 'bugs', 'cgroups', 'ci', + 'ci-dashboard', 'coding-style', 'committer-guidelines', 'compiling', -- 2.36.1
Most importantly, how to get it, how install dependencies and how to run it. Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/testtck.rst | 99 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 88 insertions(+), 11 deletions(-) diff --git a/docs/testtck.rst b/docs/testtck.rst index 51de095657..6264b40de7 100644 --- a/docs/testtck.rst +++ b/docs/testtck.rst @@ -2,6 +2,8 @@ libvirt TCK : Technology Compatibility Kit ========================================== +.. contents:: + The libvirt TCK provides a framework for performing testing of the integration between libvirt drivers, the underlying virt hypervisor technology, related operating system services and system configuration. The idea (and name) is @@ -23,15 +25,90 @@ determine the level of compatibility of their platform, and evaluate whether it will meet their needs, and get awareness of any regressions that may have occurred since a previous test run. -For more details you can look at: - -- The initial `mail from Daniel - Berrange <https://www.redhat.com/archives/libvir-list/2009-April/msg00176.html>`__ - presenting the project. -- The `page describing - VirtTCK <https://fedoraproject.org/wiki/Features/VirtTCK>`__ the inclusion of - libvirt-TCK as a Fedora Feature. - Libvirt-TCK is maintained using `a GIT -repository <https://gitlab.com/libvirt/libvirt-tck>`__, and comment, patches and -reviews are carried on the `libvir-list <contact.html>`__ development list. +repository <https://gitlab.com/libvirt/libvirt-tck>`__. GitLab is also the place +where the whole TCK development workflow (issues, merge requests, comments) +happens. + +Using TCK +--------- + +TCK can be used independently of the enviroment, i.e. both on your local host +or in a VM. We strongly recommend using a VM for the tests as TCK might affect +your current host setup, see `Running TCK`_. + +Installing dependencies +~~~~~~~~~~~~~~~~~~~~~~~ + +Since TCK is based on libvirt Perl bindings, you'll need to have the proper +version of the bindings installed for the version of libvirt you wish to test +in order to be able execute the TCK test suite successfully. Additionally, a +number of Perl dependencies will need to be installed as well, some will be +available through the system package manager and some will likely need to be +installed from CPAN (Perl's equivalent of Python's PyPI). Here's where +`libvirt-ci's <https://gitlab.com/libvirt/libvirt-ci.git>`__ lcitool can help +with preparing a test environment in a fresh VM, taking care of the +dependencies along the way: + +:: + + $ lcitool install --target fedora-36 tck-fedora36 --wait + +would get you a new Fedora 36 VM named ``tck-fedora36``. Then + +:: + + $ lcitool update tck-fedora36 libvirt,libvirt-perl,libvirt-tck+runtime + +will install all the necessary dependencies to build libvirt, the corresponding +Perl bindings and all TCK runtime dependencies to be able to execute the tests. +We also recommend executing TCK using the Avocado framework as the test harness +engine which means that you'll have to install Avocado in the test environment +as well. You can get it either from +`PyPI <https://pypi.org/project/avocado-framework/>`__ (recommended), or if +you're on Fedora you can make use of the Avocado `module <https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html#installing-from-packages>`__. +Using Avocado is not mandatory for the time being and you can skip it, but +in the future we plan on making the TCK internal coupling with Avocado tighter. + +Running TCK +~~~~~~~~~~~ + +Once you have all the dependencies installed, you can then proceed with running +as root the test suite as root (when running with Avocado): + +:: + + # avocado --config avocado.config run --tap - ./scripts/ + +from the TCK's git root. + + +If you don't want to install Avocado you can execute tests using the +``libvirt-tck`` binary directly (again, from the git root). You'll need to pass +a few options that Avocado takes care of: + +:: + + # PERL5LIB=./lib perl bin/libvirt-tck -c <path_to_config> --force ./scripts + +Running with the ``--force`` argument is not necessary and you can safely omit +it, but it becomes useful if you need to interrupt a test run for some +reason. In such case using ``--force`` ensures the first thing TCK does before +running any tests is that it will clean up all resources from the previous test +run which may have been left behind if you had interrupted the previous TCK's +execution. + +Note that running with root privileges is necessary since some tests need +access to system resources or configs. This, along with the fact that some +tests might affect the host system are good reasons to consider using a test VM +as described above. + +Contributing a test +------------------- + +We'd appreciate if you provided a functional test case whenever you're adding a +new feature or fixing a bug in libvirt with the only complication being that +in case you're adding a new public API then a Perl binding will have to be +introduced first. After that, the best way to start is looking at some existing +tests, copy-pasting one that fits your scenario the best and tweak the +remaining bits. -- 2.36.1
Since running our functional test suite in GitLab cannot make use of the shared resources it makes sense to document the process of adding own HW to run the custom libvirt executor that powers the integration suite. This article will likely make even more sense in the future with GitLab severely cutting down on shared CI resources. Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/ci-runners.rst | 86 +++++++++++++++++++++++++++++++++++++++++++++ docs/meson.build | 1 + 2 files changed, 87 insertions(+) create mode 100644 docs/ci-runners.rst diff --git a/docs/ci-runners.rst b/docs/ci-runners.rst new file mode 100644 index 0000000000..fd5fdd121a --- /dev/null +++ b/docs/ci-runners.rst @@ -0,0 +1,86 @@ +GitLab CI Custom (Specific) Runners +=================================== + +.. contents:: + +GitLab's CI allows additional machines to be added to the project's or group's +pool of runners (a runner is a machine running the GitLab's +`gitlab-runner <https://gitlab.com/gitlab-org/gitlab-runner/>`__ agent service). +Upon registering the runner the runner will then be ready accepting CI jobs +depending on the pipeline configuration. Unlike the shared runners provided +directly by GitLab's hosted SaaS specific runners are only used within the +project/group which they were registered to, so you don't need to worry about +forks burning CPU cycles on your precious HW resources. + +Understandably, we respect your decision to keep your runners only visible to +your fork, but for the sake of the community we'd appreciate if you decided to +register your runner with the upstream libvirt project instead. As we're only +interested in running upstream test workloads (which you can even help +defining) maintenance and security of the HW is your own responsibility and so +we can promise to never ask for physical or remote access to the machine. + +Machine Setup Howto +------------------- + +The following sections will guide you through the necessary setup of your +specific GitLab runners. + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that is supposed +to run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears-like icon at the bottom of the left hand side + vertical toolbar), then + * CI/CD, then + * Runners, and click on the *Expand* button, then + * Under *Set up a specific Runner manually*, look for the value under + *And this registration token:* + +Note that in order to register a runner with the upstream libvirt project +you'll need to work with the project maintainers to successfully register your +machine. + +Following the `registration <https://docs.gitlab.com/runner/register/>`__ +process, it's necessary to configure the runner tags, and optionally other +configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the *Expand* button, then + * *Runners activated for this project*, then + * Click on the *Edit* icon (next to the *Lock* Icon) + +Don't forget to add a tag to your runner as these are used to route specific +jobs to specific runners, e.g. if a job in ``ci/integration.yml`` looked like +this :: + + centos-stream-9-tests: + ... + variables: + # needed by libvirt-gitlab-executor + DISTRO: centos-stream-9 + # can be overridden in forks to set a different runner tag + LIBVIRT_CI_INTEGRATION_RUNNER_TAG: my-vm-host + tags: + - $LIBVIRT_CI_INTEGRATION_RUNNER_TAG + +it would mean that the CentOS Stream 9 job would only be scheduled on runners +bearing the 'my-vm-host' tag. + +Running integration tests +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Libvirt's integration tests run in a nested virtualization environment. So, if +you wish to run integration tests on your bare-metal machine, you'll have to +make use of GitLab's +`custom executor <https://docs.gitlab.com/runner/executors/custom.html>`__ +feature which allows you to provision any kind of environment for a workload to +run - in libvirt's case - a virtual machine. If you need any help with creating +VM template images ready to run libvirt's integration test suite, have a look +at the `libvirt-gitlab-executor <https://gitlab.com/libvirt/libvirt-custom-executor>`__ +project which encapsulates provisioning, execution, and teardown of the +virtualized environments in a single tool. diff --git a/docs/meson.build b/docs/meson.build index fd9df047cb..0c7c54a75f 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -35,6 +35,7 @@ docs_rst_files = [ 'cgroups', 'ci', 'ci-dashboard', + 'ci-runners', 'coding-style', 'committer-guidelines', 'compiling', -- 2.36.1
Currently we don't have much information on how testing is done in libvirt and the little we have is scattered among multiple files. This patch creates a common landing page containing all important bits about testing in libvirt, providing links to respective sections which deserve their own articles. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/meson.build | 1 + docs/testing.rst | 172 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+) create mode 100644 docs/testing.rst diff --git a/docs/meson.build b/docs/meson.build index 0c7c54a75f..6d61e18385 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -105,6 +105,7 @@ docs_rst_files = [ 'submitting-patches', 'support', 'testapi', + 'testing', 'testsuites', 'testtck', 'uri', diff --git a/docs/testing.rst b/docs/testing.rst new file mode 100644 index 0000000000..21abf42080 --- /dev/null +++ b/docs/testing.rst @@ -0,0 +1,172 @@ +======= +Testing +======= + +.. contents:: + +Different types of tests are available to libvirt developers for testing a +given libvirt release. + +Unit tests +---------- + +The unit test suite present in the source code is mainly used to test our +XML parser/formatter, QEMU command line generator, QEMU capabilities probing, +etc. It is run by developers before submitting patches upstream and is +mandatory to pass for any contribution to be accepted upstream. One can run +the test suite in the source tree with the following:: + + $ ninja test + + +Container builds +---------------- + +Technically speaking these are not tests in the common sense. However, usage of +public container images to build libvirt in predefined and widely accessible +environments makes it possible to expand our coverage across distros, +architectures, toolchain flavors and library versions and as such is a very +valuable marker when accepting upstream contributions. Therefore, it is +recommended to run libvirt builds against your changes in various containers to +either locally or by using GitLab's shared CI runners to make sure everything +runs cleanly before submitting your patches. The images themselves come from +libvirt's GitLab container registry, but this can be overriden if needed, see +below. + +Registry +~~~~~~~~ + +Libvirt project has its container registry hosted by GitLab at +``registry.gitlab.com/libvirt/libvirt`` which will automatically be +used to pull in pre-built layers. This avoids the need to build all the +containers locally using the Dockerfile recipes found in ``ci/containers/``. + + +Running container builds with GitLab CI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As long as your GitLab account has CI minutes available, pipelines will run +automatically on every branch push to your fork. + +Running container builds locally +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to run container builds locally, we have a ``helper`` script inside +the ``ci`` directory that can pull, build, and test (if applicable) changes on +your current local branch. It supports both the Docker and Podman runtimes +with an automatic selection of whichever runtime is configured on your system. +In case neither has been enabled/configured, please go through the following +prerequisites. We recommend using podman because of its daemonless architecture +and security implications (i.e. rootless container execution by default) over +Docker. + +Podman Prerequisites +~~~~~~~~~~~~~~~~~~~~ + +Install "podman" with the system package manager. + +.. code:: + + $ sudo dnf install -y podman + $ podman ps + +The last command should print an empty table, to verify the system is ready. + +Docker Prerequisites +~~~~~~~~~~~~~~~~~~~~ + +Install "docker" with the system package manager and start the Docker service +on your development machine, then make sure you have the privilege to run +Docker commands. Typically it means setting up passwordless ``sudo docker`` +command or login as root. For example: + +.. code:: + + $ sudo dnf install -y docker + $ # or `apt-get install docker` for Ubuntu, etc. + $ sudo systemctl start docker + $ sudo docker ps + +The last command should print an empty table, to verify the system is ready. + +An alternative method to set up permissions is by adding the current user to +"docker" group and making the docker daemon socket file (by default +``/var/run/docker.sock``) accessible to the group: + +.. code:: + + $ sudo groupadd docker + $ sudo usermod $USER -a -G docker + $ sudo chown :docker /var/run/docker.sock + +Note that any one of above configurations makes it possible for the user to +exploit the whole host with Docker bind mounting or other privileged +operations. So only do it on development machines. + +Examples of executing local container builds +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +All of the following examples will utilize ``helper`` script mentioned earlier +sections. Let's start with the basics - listing available container images in +the default libvirt registry: + +:: + + $ cd <libvirt_git>/ci + $ ./helper --help + $ ./helper list-images + Available x86 container images: + + ... + alpine-edge + fedora-rawhide + ... + + Available cross-compiler container images: + + ... + debian-sid-cross-s390x + fedora-rawhide-cross-mingw32 + fedora-rawhide-cross-mingw64 + ... + +Now let's say one would want to build their local libvirt changes on Alpine +Edge using their own GitLab's registry container. They'd then proceed with + +:: + + $ ci/helper build --image-prefix registry.gitlab.com/<user>/libvirt/ci- alpine-edge + +Finally, it would be nice if one could get an interactive shell inside the +test environment to debug potential build issues. This can be achieved with the +following: + +:: + + $ ci/helper shell alpine-edge + + +Integration tests +----------------- + +There are a few frameworks for writing and running functional tests in libvirt +with TCK being the one that runs in our upstream CI. + +- the `TCK test suite <testtck.html>`__ is a functional test suite implemented + using the `Perl bindings <https://search.cpan.org/dist/Sys-Virt/>`__ of + libvirt. This is the recommended framework to use for writing upstream + functional tests at the moment. You can start by cloning the + `TCK git repo <https://gitlab.com/libvirt/libvirt-tck>`__. + +- the `Avocado VT <https://github.com/avocado-framework/avocado-vt>`__ test + suite with the libvirt plugin is another framework implementing functional + testing utilizing the Avocado test framework underneath. Although written in + Python, the vast majority of the tests are exercising libvirt through the + command line client ``virsh``. + +- the `libvirt-test-API <testapi.html>`__ is also a functional test suite, but + implemented using the `Python bindings <python.html>`__ of libvirt. + Unfortunately this framework is the least recommended one as it's largely + unmaintained and may be completely deprecated in the future in favour of TCK. + You can get it by cloning the + `git repo <https://gitlab.com/libvirt/libvirt-test-API/>`__. -- 2.36.1
On Thu, Jul 14, 2022 at 05:34:29PM +0200, Erik Skultety wrote:
Currently we don't have much information on how testing is done in libvirt and the little we have is scattered among multiple files. This patch creates a common landing page containing all important bits about testing in libvirt, providing links to respective sections which deserve their own articles.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/meson.build | 1 + docs/testing.rst | 172 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+) create mode 100644 docs/testing.rst
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
The new article provides more in-depth information on testing options in libvirt. Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/docs.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/docs.rst b/docs/docs.rst index 9fd8b7c37e..b17d73743a 100644 --- a/docs/docs.rst +++ b/docs/docs.rst @@ -154,10 +154,8 @@ Project development `API extensions <api_extension.html>`__ Adding new public libvirt APIs -`Functional testing <testsuites.html>`__ - Testing libvirt with - `TCK test suite <testtck.html>`__ and - `Libvirt-test-API <testapi.html>`__ +`Testing <testing.html>`__ + Details various types of testing available for libvirt `New repo setup <newreposetup.html>`__ Procedure for configuring new git repositories for libvirt -- 2.36.1
The article was replaced with a new one in previous commit, so we don't need this one anymore. Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/meson.build | 1 - docs/testsuites.rst | 37 ------------------------------------- 2 files changed, 38 deletions(-) delete mode 100644 docs/testsuites.rst diff --git a/docs/meson.build b/docs/meson.build index 6d61e18385..0b16d64488 100644 --- a/docs/meson.build +++ b/docs/meson.build @@ -106,7 +106,6 @@ docs_rst_files = [ 'support', 'testapi', 'testing', - 'testsuites', 'testtck', 'uri', 'windows', diff --git a/docs/testsuites.rst b/docs/testsuites.rst deleted file mode 100644 index 4fc733e615..0000000000 --- a/docs/testsuites.rst +++ /dev/null @@ -1,37 +0,0 @@ -=========== -Test suites -=========== - -There is a few test suites available to developers for testing a given version -of libvirt: - -- the internal test suite: present in the source code, it is run by developers - before submitting patches upstream, it is also suggested to have it run and - pass as part of the packaging process for distributions. It is run by - launching: - - :: - - make check (libvirt 6.6.0 and older) - - :: - - ninja test (libvirt 6.7.0 and newer) - - in a source tree after compilation has finished. It doesn't really make - functional testing but checks that large portions of the code not interacting - directly with virtualization functions properly. - -- the `TCK test suite <testtck.html>`__ is a functional test suite implemented - using the `Perl bindings <https://search.cpan.org/dist/Sys-Virt/>`__ of - libvirt. It is available separately as a - `download <ftp://libvirt.org/libvirt/tck/>`__, as a - `package <https://rpmfind.net/linux/rpm2html/search.php?query=libvirt-tck>`__ - in Fedora distributions, but best is probably to get the `version from - GIT <https://gitlab.com/libvirt/libvirt-tck>`__. - -- the `libvirt-test-API <testapi.html>`__ is also a functional test suite, but - implemented using the `Python bindings <python.html>`__ of libvirt. It is - available separately as a - `download <ftp://libvirt.org/libvirt/libvirt-test-API/>`__, or directly get - the `version from GIT <https://gitlab.com/libvirt/libvirt-test-API/>`__. -- 2.36.1
It's not just strategy the master CI article talks (or will talk in the future) about. Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/docs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs.rst b/docs/docs.rst index b17d73743a..8ca0afdc1a 100644 --- a/docs/docs.rst +++ b/docs/docs.rst @@ -136,8 +136,8 @@ Project development `Project strategy <strategy.html>`__ Sets a vision for future direction & technical choices -`CI Testing <ci.html>`__ - Details of the Continuous Integration testing strategy +`CI <ci.html>`__ + Details on our Continuous Integration `Bug reports <bugs.html>`__ How and where to report bugs and request features -- 2.36.1
On Thu, Jul 14, 2022 at 05:34:32PM +0200, Erik Skultety wrote:
It's not just strategy the master CI article talks (or will talk in the future) about.
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/docs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/docs/ci.rst b/docs/ci.rst index d5282c4d25..566bc42db7 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -8,6 +8,9 @@ The libvirt project uses GitLab CI for automated testing. `Here's <ci-dashboard.html>`__ our CI dashboard which shows the current status of our pipelines. +Builds and unit tests +===================== + Linux builds and cross-compiled Windows builds happen on GitLab CI's shared runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI <https://cirrus-ci.com/>`_ jobs behind the scenes. @@ -15,27 +18,25 @@ runners, while FreeBSD and macOS coverage is achieved by triggering `Cirrus CI Most of the tooling used to build CI pipelines is maintained as part of the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci>`_ subproject. +Integration tests +================= +Integration tests in our CI pipelines require dedicated HW which is not +available to forks, see `GitLab CI Custom Runners <ci-runners.html>`__. +Therefore, in order to execute the integration tests as part of your libvirt +fork's GitLab CI you'll need to provide your own runner. You'll also need to +set a few CI variables to run the integration tests as part of the CI pipeline, +see below. +GitLab CI variables +------------------- +* ``LIBVIRT_CI_INTEGRATION`` - enables integration test runs manually or in forks +* ``LIBVIRT_CI_INTEGRATION_RUNNER_TAG`` - overrides the upstream runner tag on the +Retrieving test logs +-------------------- - - - - - - - - - - - - - - - - - - - +In case the integration test suite fails in our CI pipelines, a job artifact is +generated containing Avocado logs, libvirt debug logs, and the latest traceback +(if one was produced during a daemon's execution). -- 2.36.1
On Thu, Jul 14, 2022 at 05:34:33PM +0200, Erik Skultety wrote:
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/ci.rst b/docs/ci.rst index 566bc42db7..efb1ea295e 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -40,3 +40,12 @@ Retrieving test logs In case the integration test suite fails in our CI pipelines, a job artifact is generated containing Avocado logs, libvirt debug logs, and the latest traceback (if one was produced during a daemon's execution). + +Adding new OS platforms OR build pre-requisites +=============================================== + +Since all of the Dockerfiles libvirt uses for CI have been generated by ``lcitool`` +provided by the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci.git>`__ project, +most relevant changes will need to be introduced to ``lcitool`` first. Please +follow the instructions outlined +`here <https://gitlab.com/libvirt/libvirt-ci/-/blob/master/docs/platforms_and_mappings.rst>`__ -- 2.36.1
On Thu, Jul 14, 2022 at 05:34:34PM +0200, Erik Skultety wrote:
Signed-off-by: Erik Skultety <eskultet@redhat.com> --- docs/ci.rst | 9 +++++++++ 1 file changed, 9 insertions(+)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
This is just a glue to the testing article introduced in previous commits. Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/ci.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/ci.rst b/docs/ci.rst index efb1ea295e..ee85018c49 100644 --- a/docs/ci.rst +++ b/docs/ci.rst @@ -49,3 +49,10 @@ provided by the `libvirt-ci <https://gitlab.com/libvirt/libvirt-ci.git>`__ proje most relevant changes will need to be introduced to ``lcitool`` first. Please follow the instructions outlined `here <https://gitlab.com/libvirt/libvirt-ci/-/blob/master/docs/platforms_and_mappings.rst>`__ + + +Running CI workloads locally +============================ + +If you're interested in running the CI test workloads locally, please read +our `testing <testing.html>`__ guide. -- 2.36.1
participants (2)
-
Daniel P. Berrangé -
Erik Skultety