- Devel - libvirt List Archives

[libvirt PATCH] docs: add info about factors affecting CPU compatibility
by Daniel P. Berrangé 12 Jul '22

12 Jul '22

While libvirt solves most of the problem of ensuring compatibility, when there is incompatibility it can be hard for users to track down the cause. Everything knows to check the physical CPU model, but there are a surprisingly large number of other factors influencing compatibility. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- docs/drvqemu.rst | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/docs/drvqemu.rst b/docs/drvqemu.rst index c174bdb4cf..3344df4fb6 100644 --- a/docs/drvqemu.rst +++ b/docs/drvqemu.rst @@ -416,6 +416,86 @@ following command should be run as root, prior to starting libvirtd libvirt will then place each virtual machine in a cgroup at ``/dev/cgroup/libvirt/qemu/$VMNAME/`` + +Live migration compatibility +---------------------------- + +Many factors can affect the ability to live migrate a guest between a pair +of hosts. It is critical that when QEMU is started on the destination host, +the exposed guest machine ABI matches what was exposed by the existing QEMU +process on the source host. To facilitate this, when libvirt receives a +guest configuration document, it will attempt to expand any features that +were not specified, to ensure a stable guest machine ABI. Mostly this involves +adding address information to all devices, and adding controllers to attach +the devices to. + +Certain features that affect the guest ABI, however, may only be known at the +time the guest is started and can be influenced by features of the host OS +and its hardware. This means that even if the guest XML configuration is the +same, it may still be impossible to migrate the guest between two hosts. + +Migration CPU model compatibility +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most common problems with migration compatibility surround the use of +the guest CPU ``host-model`` or ``host-passthrough`` modes. Both of these +modes attempt to expose the full host CPU featureset to the guest. The +``host-model`` mode attempts to expose as many features as possible while +retaining the ability to accurately check compatibility between hosts prior +to migration running. The ``host-passthrough`` mode attempts to expose the +host CPU as precisely as possible, but with the cost that it is not possible +for libvirt to check compatibility prior to migration. + +If using ``host-model`` the target host hardware and software deployment +must expose a superset of the features of the source host CPU. If using +``host-passthrough`` the target host CPU and software deployment must +always expose a superset of the fetures, however, it is further strongly +recommended that the source and destination hosts be identical in every +way. + +In both cases, there are a number of factors that will influence the CPU +features available to the guest + +* **Physical CPU model** - the core constraint on what features are available. + Check ``/proc/cpuinfo`` for CPU model name. +* **Firmware revision (BIOS/UEFI/etc)** - firmware updates may bundle microcode + updates which arbitrarily add or remove CPU features, typically in response + to new hardware vulnerabilities. Check ``dmidecode`` for details on ``x86`` + and ``aarch64`` platforms for firmware version, and ``/proc/cpuinfo`` for + associated microcode version (if not updated by the OS). +* **Firmware settings** - certain firmware settings can affect accessibility of + features. For example, turning on/off SMT/HT not only affects the number + of logical CPUs available to the OS, but can indirectly influence other + factors such as the number of performance counters available for use. Check + the firmware specific configuration interface. +* **Host kernel version** - the host kernel software version may have a + need to block certain physical CPU features from use in the guest. It can + also emulate certain features that may not exist in the silicon, for example, + x2apic. Check ``uname -r`` output for kernel version. +* **Host kernel settings** - the kernel command line options can be used to + block certain physical CPU features from use in the guest, for example, + ``tsx=off``, ``l1tf=...`` or ``nosmt``. Check ``/proc/cmdline`` and + ``/etc/modprobe.d/*.conf``. +* **microcode update version** - while the firmware will load the initial + microcode in to the CPU, the OS may ship packages providing newer microcode + updates since these can be deployed on a more timely manner than firmware + updates. These updates can arbitrarily load add or remove CPU features. + Check ``/proc/cpuinfo`` for microcode version. +* **QEMU version** - even when the kernel supports exposing a CPU feature to + the guest, an update in the QEMU emulator version will be required to unlock + its usage with a guest, except with ``host-passthrough``. Check the output + of ``$QEMU -version``. +* **libvirt version** - even when the kernel and QEMU support exposing a CPU + feature to the guest, an update in the libvirt version will be required to + unlock its usage with a guest, except with ``host-passthrough``. Check + ``virsh version``. + +The ``virsh capabilities`` output will provide information on the high level +CPU model, its features, microcode version. Most of the time this will provide +enough information to know whether the CPUs of two hosts will be compatible. +If there are unexpected differences though, checking the above list of +influencing factors can reveal where the difference arises from. + Import and export of libvirt domain XML configs ----------------------------------------------- -- 2.36.1

1 0

[PATCH] remote: conf: Mention that 'virtproxyd' doesn't require the '--listen' flag
by Peter Krempa 12 Jul '22

12 Jul '22

'virtproxyd' doesn't have the '--listen' parameter and it's enough just to enable the corresponding option in the config. The config file is generic for all the daemons and we just omit/adjust some sections. Adding a separate mechanism to omit the note about the '--listen' parameter would be overkill so mention explicitly that it's required only for libvirtd and not virtproxyd. The section is omitted for other daemons. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2094641 Signed-off-by: Peter Krempa <pkrempa(a)redhat.com> --- src/remote/libvirtd.conf.in | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/src/remote/libvirtd.conf.in b/src/remote/libvirtd.conf.in index defe6b83d0..929d3922e9 100644 --- a/src/remote/libvirtd.conf.in +++ b/src/remote/libvirtd.conf.in @@ -8,8 +8,10 @@ # # Flag listening for secure TLS connections on the public TCP/IP port. -# NB, must pass the --listen flag to the @DAEMON_NAME@ process for this to -# have any effect. +# +# To enable listening sockets with the 'libvirtd' daemon it's also required to +# pass the --listen flag on the commandline of the daemon. +# This is not needed with 'virtproxyd'. # # This setting is not required or honoured if using systemd socket # activation. @@ -21,8 +23,10 @@ #listen_tls = 0 # Listen for unencrypted TCP connections on the public TCP/IP port. -# NB, must pass the --listen flag to the @DAEMON_NAME@ process for this to -# have any effect. +# +# To enable listening sockets with the 'libvirtd' daemon it's also required to +# pass the --listen flag on the commandline of the daemon. +# This is not needed with 'virtproxyd'. # # This setting is not required or honoured if using systemd socket # activation. -- 2.36.1

2 1

[(pushed) PATCH] docs: formatstorage: Format XML namespace definition as code block
by Peter Krempa 12 Jul '22

12 Jul '22

Signed-off-by: Peter Krempa <pkrempa(a)redhat.com> --- Trivial. docs/formatstorage.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/formatstorage.rst b/docs/formatstorage.rst index 83d7d141ac..a8149e450c 100644 --- a/docs/formatstorage.rst +++ b/docs/formatstorage.rst @@ -470,9 +470,9 @@ option in libvirt, and thus should never be used in production. options for the mount command via the "-o" option for the ``fs`` or ``netfs`` type storage pools. In order to designate that the Storage Pool will be using the mechanism, the ``pool`` element must be modified to provide the XML - namespace attribute syntax as follows: + namespace attribute syntax as follows:: - xmlns:fs='http://libvirt.org/schemas/storagepool/fs/1.0' + xmlns:fs='http://libvirt.org/schemas/storagepool/fs/1.0' The ``fs:mount_opts`` defines the mount options by specifying multiple ``fs:option`` subelements with the attribute ``name`` specifying the mount -- 2.36.1

1 0

[PATCH] ci: Regenerate files with new lcitool
by Peter Krempa 11 Jul '22

11 Jul '22

Apart from other changes this fixes failures with builds on FreeBSD. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com> --- Note that this is based on lcitool with the following MR merged: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/290 which is necessary to fix FreeBSD12: https://gitlab.com/pipo.sk/libvirt/-/pipelines/584873900 ci/cirrus/freebsd-12.vars | 2 +- ci/cirrus/freebsd-13.vars | 2 +- ci/containers/centos-stream-8.Dockerfile | 1 + ci/containers/centos-stream-9.Dockerfile | 14 ++++-- ci/gitlab.yml | 22 +++++++++ ci/gitlab/build-templates.yml | 12 +++-- ci/gitlab/builds.yml | 60 ++++++++++++------------ ci/gitlab/container-templates.yml | 8 ++-- ci/gitlab/containers.yml | 43 ++++++++++------- 9 files changed, 105 insertions(+), 59 deletions(-) diff --git a/ci/cirrus/freebsd-12.vars b/ci/cirrus/freebsd-12.vars index e8b06b32e1..b3b3d7a187 100644 --- a/ci/cirrus/freebsd-12.vars +++ b/ci/cirrus/freebsd-12.vars @@ -11,6 +11,6 @@ MAKE='/usr/local/bin/gmake' NINJA='/usr/local/bin/ninja' PACKAGING_COMMAND='pkg' PIP3='/usr/local/bin/pip-3.8' -PKGS='augeas bash-completion ca_root_nss ccache codespell cppi curl cyrus-sasl diffutils diskscrub fusefs-libs gettext git glib gmake gnugrep gnutls gsed libpcap libpciaccess libssh libssh2 libxml2 libxslt meson ninja perl5 pkgconf polkit py38-docutils py38-flake8 python3 qemu readline yajl' +PKGS='augeas bash-completion ca_root_nss ccache codespell cppi curl cyrus-sasl diffutils diskscrub fusefs-libs gettext git glib gmake gnugrep gnutls gsed libpcap libpciaccess libssh libssh2 libxml2 libxslt meson ninja perl5 pkgconf polkit py39-docutils py39-flake8 python3 qemu readline yajl' PYPI_PKGS='' PYTHON='/usr/local/bin/python3' diff --git a/ci/cirrus/freebsd-13.vars b/ci/cirrus/freebsd-13.vars index e8b06b32e1..b3b3d7a187 100644 --- a/ci/cirrus/freebsd-13.vars +++ b/ci/cirrus/freebsd-13.vars @@ -11,6 +11,6 @@ MAKE='/usr/local/bin/gmake' NINJA='/usr/local/bin/ninja' PACKAGING_COMMAND='pkg' PIP3='/usr/local/bin/pip-3.8' -PKGS='augeas bash-completion ca_root_nss ccache codespell cppi curl cyrus-sasl diffutils diskscrub fusefs-libs gettext git glib gmake gnugrep gnutls gsed libpcap libpciaccess libssh libssh2 libxml2 libxslt meson ninja perl5 pkgconf polkit py38-docutils py38-flake8 python3 qemu readline yajl' +PKGS='augeas bash-completion ca_root_nss ccache codespell cppi curl cyrus-sasl diffutils diskscrub fusefs-libs gettext git glib gmake gnugrep gnutls gsed libpcap libpciaccess libssh libssh2 libxml2 libxslt meson ninja perl5 pkgconf polkit py39-docutils py39-flake8 python3 qemu readline yajl' PYPI_PKGS='' PYTHON='/usr/local/bin/python3' diff --git a/ci/containers/centos-stream-8.Dockerfile b/ci/containers/centos-stream-8.Dockerfile index 82cbd7cde7..1614cd9b56 100644 --- a/ci/containers/centos-stream-8.Dockerfile +++ b/ci/containers/centos-stream-8.Dockerfile @@ -11,6 +11,7 @@ RUN dnf distro-sync -y && \ dnf config-manager --set-enabled -y powertools && \ dnf install -y centos-release-advanced-virtualization && \ dnf install -y epel-release && \ + dnf install -y epel-next-release && \ dnf install -y \ audit-libs-devel \ augeas \ diff --git a/ci/containers/centos-stream-9.Dockerfile b/ci/containers/centos-stream-9.Dockerfile index 34ff3a06eb..5d0b5ffc6c 100644 --- a/ci/containers/centos-stream-9.Dockerfile +++ b/ci/containers/centos-stream-9.Dockerfile @@ -9,14 +9,14 @@ FROM quay.io/centos/centos:stream9 RUN dnf distro-sync -y && \ dnf install 'dnf-command(config-manager)' -y && \ dnf config-manager --set-enabled -y crb && \ - dnf install -y \ - https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \ - https://dl.fedoraproject.org/pub/epel/epel-next-release-latest-9.noarch.rpm && \ + dnf install -y epel-release && \ + dnf install -y epel-next-release && \ dnf install -y \ audit-libs-devel \ augeas \ bash-completion \ ca-certificates \ + ccache \ clang \ cpp \ cyrus-sasl-devel \ @@ -50,6 +50,7 @@ RUN dnf distro-sync -y && \ librbd-devel \ libselinux-devel \ libssh-devel \ + libssh2-devel \ libtirpc-devel \ libwsman-devel \ libxml2 \ @@ -81,9 +82,14 @@ RUN dnf distro-sync -y && \ yajl-devel && \ dnf autoremove -y && \ dnf clean all -y && \ - rpm -qa | sort > /packages.txt + rpm -qa | sort > /packages.txt && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/clang && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc ENV LANG "en_US.UTF-8" ENV MAKE "/usr/bin/make" ENV NINJA "/usr/bin/ninja" ENV PYTHON "/usr/bin/python3" +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/gitlab.yml b/ci/gitlab.yml index cab4e6074c..ea0f847789 100644 --- a/ci/gitlab.yml +++ b/ci/gitlab.yml @@ -4,6 +4,28 @@ # # https://gitlab.com/libvirt/libvirt-ci + +# Variables that can be set to control the behaviour of +# pipelines that are run +# +# - RUN_ALL_CONTAINERS - build all containers +# even if they don't have any changes detected +# +# These can be set as git push options +# +# $ git push -o ci.variable=RUN_ALL_CONTAINERS=1 +# +# Aliases can be set for common usage +# +# $ git config --local alias.push-all-ctr "push -o ci.variable=RUN_ALL_CONTAINERS=1" +# +# Allowing the less verbose invokation +# +# $ git push-all-ctr +# +# Pipeline variables can also be set in the repository +# pipeline config globally, or set against scheduled pipelines + include: - local: '/ci/gitlab/container-templates.yml' - local: '/ci/gitlab/build-templates.yml' diff --git a/ci/gitlab/build-templates.yml b/ci/gitlab/build-templates.yml index 20d22914c2..605788dab9 100644 --- a/ci/gitlab/build-templates.yml +++ b/ci/gitlab/build-templates.yml @@ -9,7 +9,9 @@ image: $CI_REGISTRY_IMAGE/ci-$NAME:latest stage: builds rules: - - if: '$LIBVIRT_JOB_OPTIONAL' + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: never + - if: '$JOB_OPTIONAL' when: manual allow_failure: true - when: on_success @@ -19,7 +21,9 @@ image: $CI_REGISTRY_IMAGE/ci-$NAME-cross-$CROSS:latest stage: builds rules: - - if: '$LIBVIRT_JOB_OPTIONAL' + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: never + - if: '$JOB_OPTIONAL' when: manual allow_failure: true - when: on_success @@ -52,9 +56,9 @@ - cat ci/cirrus/$NAME.yml - cirrus-run -v --show-build-log always ci/cirrus/$NAME.yml rules: - - if: "$CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null" + - if: '$CIRRUS_GITHUB_REPO == null || $CIRRUS_API_TOKEN == null' when: never - - if: '$LIBVIRT_JOB_OPTIONAL' + - if: '$JOB_OPTIONAL' when: manual allow_failure: true - when: on_success diff --git a/ci/gitlab/builds.yml b/ci/gitlab/builds.yml index 3ce110d39e..443c1ac299 100644 --- a/ci/gitlab/builds.yml +++ b/ci/gitlab/builds.yml @@ -14,7 +14,7 @@ x86_64-almalinux-8: optional: true allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: almalinux-8 RPM: skip @@ -48,7 +48,7 @@ x86_64-alpine-315: optional: true allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: alpine-315 @@ -107,7 +107,7 @@ x86_64-debian-11: optional: true allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -223,7 +223,7 @@ x86_64-ubuntu-2204: optional: true allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: ubuntu-2204 @@ -251,7 +251,7 @@ aarch64-debian-10: allow_failure: true variables: CROSS: aarch64 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -263,7 +263,7 @@ armv6l-debian-10: allow_failure: true variables: CROSS: armv6l - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -286,7 +286,7 @@ i686-debian-10: allow_failure: true variables: CROSS: i686 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -298,7 +298,7 @@ mips-debian-10: allow_failure: true variables: CROSS: mips - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -310,7 +310,7 @@ mips64el-debian-10: allow_failure: true variables: CROSS: mips64el - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -322,7 +322,7 @@ mipsel-debian-10: allow_failure: true variables: CROSS: mipsel - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -334,7 +334,7 @@ ppc64le-debian-10: allow_failure: true variables: CROSS: ppc64le - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -346,7 +346,7 @@ s390x-debian-10: allow_failure: true variables: CROSS: s390x - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-10 @@ -358,7 +358,7 @@ aarch64-debian-11: allow_failure: false variables: CROSS: aarch64 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -370,7 +370,7 @@ armv6l-debian-11: allow_failure: false variables: CROSS: armv6l - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -382,7 +382,7 @@ armv7l-debian-11: allow_failure: false variables: CROSS: armv7l - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -394,7 +394,7 @@ i686-debian-11: allow_failure: false variables: CROSS: i686 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -406,7 +406,7 @@ mips64el-debian-11: allow_failure: false variables: CROSS: mips64el - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -418,7 +418,7 @@ mipsel-debian-11: allow_failure: false variables: CROSS: mipsel - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -430,7 +430,7 @@ ppc64le-debian-11: allow_failure: false variables: CROSS: ppc64le - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -442,7 +442,7 @@ s390x-debian-11: allow_failure: false variables: CROSS: s390x - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11 @@ -454,7 +454,7 @@ aarch64-debian-sid: allow_failure: false variables: CROSS: aarch64 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -466,7 +466,7 @@ armv6l-debian-sid: allow_failure: false variables: CROSS: armv6l - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -478,7 +478,7 @@ armv7l-debian-sid: allow_failure: false variables: CROSS: armv7l - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -490,7 +490,7 @@ i686-debian-sid: allow_failure: false variables: CROSS: i686 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -502,7 +502,7 @@ mips64el-debian-sid: allow_failure: false variables: CROSS: mips64el - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -514,7 +514,7 @@ mipsel-debian-sid: allow_failure: false variables: CROSS: mipsel - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -526,7 +526,7 @@ ppc64le-debian-sid: allow_failure: false variables: CROSS: ppc64le - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -538,7 +538,7 @@ s390x-debian-sid: allow_failure: false variables: CROSS: s390x - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid @@ -550,7 +550,7 @@ mingw32-fedora-36: allow_failure: false variables: CROSS: mingw32 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: fedora-36 @@ -584,7 +584,7 @@ mingw64-fedora-rawhide: allow_failure: true variables: CROSS: mingw64 - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: fedora-rawhide diff --git a/ci/gitlab/container-templates.yml b/ci/gitlab/container-templates.yml index c71d03c759..0e52f147bb 100644 --- a/ci/gitlab/container-templates.yml +++ b/ci/gitlab/container-templates.yml @@ -10,7 +10,7 @@ # - Push to default branch: # -> rebuild if dockerfile changed, no cache # - Otherwise -# -> rebuild if LIBVIRT_CI_CONTAINERS=1, no cache, +# -> rebuild if RUN_ALL_CONTAINERS=1, no cache, # to pick up new published distro packages or # recover from deleted tag # @@ -40,16 +40,18 @@ after_script: - docker logout rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: never - if: '$CI_PROJECT_NAMESPACE == "libvirt" && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' when: on_success changes: - ci/gitlab/container-templates.yml - ci/containers/$NAME.Dockerfile - - if: '$CI_PROJECT_NAMESPACE == "libvirt" && $LIBVIRT_CI_CONTAINERS == "1"' + - if: '$CI_PROJECT_NAMESPACE == "libvirt" && $RUN_ALL_CONTAINERS == "1"' when: on_success - if: '$CI_PROJECT_NAMESPACE == "libvirt"' when: never - - if: '$LIBVIRT_JOB_OPTIONAL' + - if: '$JOB_OPTIONAL' when: manual allow_failure: true - when: on_success diff --git a/ci/gitlab/containers.yml b/ci/gitlab/containers.yml index 7cdf3eb9a0..3835a19216 100644 --- a/ci/gitlab/containers.yml +++ b/ci/gitlab/containers.yml @@ -25,6 +25,7 @@ x86_64-alpine-315-container: extends: .container_job allow_failure: false variables: + JOB_OPTIONAL: 1 NAME: alpine-315 @@ -126,6 +127,7 @@ aarch64-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-aarch64 @@ -133,6 +135,7 @@ armv6l-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-armv6l @@ -147,6 +150,7 @@ i686-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-i686 @@ -154,6 +158,7 @@ mips-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-mips @@ -161,6 +166,7 @@ mips64el-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-mips64el @@ -168,6 +174,7 @@ mipsel-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-mipsel @@ -175,6 +182,7 @@ ppc64le-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-ppc64le @@ -182,6 +190,7 @@ s390x-debian-10-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: debian-10-cross-s390x @@ -189,7 +198,7 @@ aarch64-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-aarch64 @@ -197,7 +206,7 @@ armv6l-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-armv6l @@ -205,7 +214,7 @@ armv7l-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-armv7l @@ -213,7 +222,7 @@ i686-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-i686 @@ -221,7 +230,7 @@ mips64el-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-mips64el @@ -229,7 +238,7 @@ mipsel-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-mipsel @@ -237,7 +246,7 @@ ppc64le-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-ppc64le @@ -245,7 +254,7 @@ s390x-debian-11-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-11-cross-s390x @@ -253,7 +262,7 @@ aarch64-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-aarch64 @@ -261,7 +270,7 @@ armv6l-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-armv6l @@ -269,7 +278,7 @@ armv7l-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-armv7l @@ -277,7 +286,7 @@ i686-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-i686 @@ -285,7 +294,7 @@ mips64el-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-mips64el @@ -293,7 +302,7 @@ mipsel-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-mipsel @@ -301,7 +310,7 @@ ppc64le-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-ppc64le @@ -309,7 +318,7 @@ s390x-debian-sid-container: extends: .container_job allow_failure: false variables: - LIBVIRT_JOB_OPTIONAL: 1 + JOB_OPTIONAL: 1 NAME: debian-sid-cross-s390x @@ -317,6 +326,7 @@ mingw32-fedora-36-container: extends: .container_job allow_failure: false variables: + JOB_OPTIONAL: 1 NAME: fedora-36-cross-mingw32 @@ -338,4 +348,5 @@ mingw64-fedora-rawhide-container: extends: .container_job allow_failure: true variables: + JOB_OPTIONAL: 1 NAME: fedora-rawhide-cross-mingw64 -- 2.36.1

2 1

[PATCH] vircpi: Add PCIe 5.0 and 6.0 link speeds
by Michal Privoznik 11 Jul '22

11 Jul '22

The PCIe 5.0 and PCIe 6.0 standards define new link speeds: 32GT/s and 64GT/s, respectively. Update our internal enum to include these new speeds. Otherwise we format incorrect XML: <pci-express> <link validity='cap' port='0' speed='(null)' width='16'/> <link validity='sta' speed='16' width='16'/> </pci-express> Like all "good" specifications, these are also locked behind a login portal. But we can look at pciutils' source code: [1] and [2]. 1: https://git.kernel.org/pub/scm/utils/pciutils/pciutils.git/commit/ls-caps.c… 2: https://git.kernel.org/pub/scm/utils/pciutils/pciutils.git/commit/ls-caps.c… Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2105231 Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- src/util/virpci.c | 2 +- src/util/virpci.h | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/util/virpci.c b/src/util/virpci.c index 03d1d7b074..7800966963 100644 --- a/src/util/virpci.c +++ b/src/util/virpci.c @@ -45,7 +45,7 @@ VIR_LOG_INIT("util.pci"); VIR_ENUM_IMPL(virPCIELinkSpeed, VIR_PCIE_LINK_SPEED_LAST, - "", "2.5", "5", "8", "16", + "", "2.5", "5", "8", "16", "32", "64" ); VIR_ENUM_IMPL(virPCIStubDriver, diff --git a/src/util/virpci.h b/src/util/virpci.h index b9b9cd7b34..4d9193f24e 100644 --- a/src/util/virpci.h +++ b/src/util/virpci.h @@ -83,6 +83,8 @@ typedef enum { VIR_PCIE_LINK_SPEED_5, VIR_PCIE_LINK_SPEED_8, VIR_PCIE_LINK_SPEED_16, + VIR_PCIE_LINK_SPEED_32, + VIR_PCIE_LINK_SPEED_64, VIR_PCIE_LINK_SPEED_LAST } virPCIELinkSpeed; -- 2.35.1

2 1

[PATCH] virsh: Require --xpath for dumpxml
by Michal Privoznik 08 Jul '22

08 Jul '22

Historically, the dumpxml command reject any unknown arguments, for instance: virsh dumpxml fedora xxx However, after v8.5.0-rc1~31 the second argument ('xxx') is treated as an XPath, but it's not that clearly visible. Therefore, require the --xpath switch, like this: virsh dumpxml fedora --xpath xxx Yes, this breaks already released virsh, but I think we can argue that the pool of users of this particular function is very small. The sooner we do this change, the better. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2103524 Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- tools/virsh-domain.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 43034f2f81..a181ae005f 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -10461,6 +10461,7 @@ static const vshCmdOptDef opts_dumpxml[] = { }, {.name = "xpath", .type = VSH_OT_STRING, + .flags = VSH_OFLAG_REQ_OPT, .completer = virshCompleteEmpty, .help = N_("xpath expression to filter the XML document") }, -- 2.35.1

3 3

[PATCH] qemu_monitor_json: Implement logic for setting iothread.thread-pool-{min, max}
by Michal Privoznik 07 Jul '22

07 Jul '22

When virDomainSetIOThreadParams() API is called, well its QEMU impl: qemuDomainSetIOThreadParams() then typed params are parsed by qemuDomainIOThreadParseParams() into this qemuMonitorIOThreadInfo struct. In the struct we have a <int, bool> pair for every IOThread attribute we can tune through monitor. The struct is then passed to qemuMonitorJSONSetIOThread() which looks at the bool and if set then the corresponding attribute is set to given value. Each attribute is thus changed in a separate call. While this works for attributes independent of each other ("poll-max-ns", "poll-grow", "poll-shrink"), it does not always work for the other attributes ("thread-pool-min" and "thread-pool-max"). The limitation here is that the lower boundary (minimum) has to be lower (or equal to) the upper boundary (maximum) at all times. This means, that in some cases we might need to set attributes in reversed order to meet the constraint. Resolves: https://gitlab.com/libvirt/libvirt/-/issues/339 Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- src/qemu/qemu_monitor_json.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 3aad2ab212..80696de731 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -7428,6 +7428,7 @@ qemuMonitorJSONSetIOThread(qemuMonitor *mon, { g_autofree char *path = NULL; qemuMonitorJSONObjectProperty prop; + bool setMaxFirst = false; path = g_strdup_printf("/objects/iothread%u", iothreadInfo->iothread_id); @@ -7443,8 +7444,32 @@ qemuMonitorJSONSetIOThread(qemuMonitor *mon, VIR_IOTHREAD_SET_PROP("poll-max-ns", poll_max_ns); VIR_IOTHREAD_SET_PROP("poll-grow", poll_grow); VIR_IOTHREAD_SET_PROP("poll-shrink", poll_shrink); - VIR_IOTHREAD_SET_PROP("thread-pool-min", thread_pool_min); - VIR_IOTHREAD_SET_PROP("thread-pool-max", thread_pool_max); + + if (iothreadInfo->set_thread_pool_min && + iothreadInfo->set_thread_pool_max) { + int curr_max = -1; + + /* By default, the minimum is set first, followed by the maximum. But + * if the current maximum is below the minimum we want to set we need + * to set the maximum first. Otherwise would get an error because we + * would be attempting to shift minimum above maximum. */ + prop.type = QEMU_MONITOR_OBJECT_PROPERTY_INT; + if (qemuMonitorJSONGetObjectProperty(mon, path, + "thread-pool-max", &prop) < 0) + return -1; + curr_max = prop.val.iv; + + if (curr_max < iothreadInfo->thread_pool_min) + setMaxFirst = true; + } + + if (setMaxFirst) { + VIR_IOTHREAD_SET_PROP("thread-pool-max", thread_pool_max); + VIR_IOTHREAD_SET_PROP("thread-pool-min", thread_pool_min); + } else { + VIR_IOTHREAD_SET_PROP("thread-pool-min", thread_pool_min); + VIR_IOTHREAD_SET_PROP("thread-pool-max", thread_pool_max); + } #undef VIR_IOTHREAD_SET_PROP -- 2.35.1

2 1

[PATCH RFC v2 0/3] qemu: add an API for "query-stats" QMP command
by Amneesh Singh 07 Jul '22

07 Jul '22

QEMU is gaining introspectable statistics which can be queried via the "query-stats" QMP command. This patchset aims to add an API for the same. The returned JSON for "query-stats" is an array of objects containing their own respective array of statistics. Patch 1 adds the API which returns the deserialized JSON in the form of a GPtrArray of GHashTables. Patch 2 adds the "query-stats" to QEMU capabilities. Patch 3 uses the API to query the halt poll success time and the halt poll failure time. v1 -> v2 ======== I have been tinkering with the v1 patchset and have rewrote the v2 patches a couple of times. I believe the current patchset is still not perfect and would appreciate some reviews. I have another patch or two written but they do not make any significant changes to the current patchset. [1/3]: - use virBitmap instead of an array of strings for statistics. - add enums for the stat names and add qemuMonitorQueryStatsNameTypeToString to switch between the "ToString" functions based on the target type. - change qemuMonitorQueryStatsProviderNew to a variadic function that takes stat enum values with the sentinel value being -1. [2/3]: - No changes [3/3]: - Add relevant monitor related checks to check if the domain is active. - Acquire and release qemuMonitorObj lock before and after calling qemuMonitorQueryStats respectively. - Add the check for privileged access. Relevant QEMU patches can be found here: https://lore.kernel.org/all/20220530150714.756954-1-pbonzini@redhat.com/ This patchset is part of the 2022 GSOC contributor project. Amneesh Singh (3): qemu_monitor: add qemuMonitorQueryStats qemu_capabilities: add "query-stats" QMP command to the QEMU capabilities qemu_driver: use qemuMonitorQueryStats to extract halt poll time src/qemu/qemu_capabilities.c | 2 + src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_driver.c | 70 ++++++++++++++++--- src/qemu/qemu_monitor.c | 106 ++++++++++++++++++++++++++++ src/qemu/qemu_monitor.h | 56 +++++++++++++++ src/qemu/qemu_monitor_json.c | 130 +++++++++++++++++++++++++++++++++++ src/qemu/qemu_monitor_json.h | 6 ++ 7 files changed, 363 insertions(+), 8 deletions(-) -- 2.36.1

2 6

[PATCH 00/11] Test case cleanups
by Peter Krempa 07 Jul '22

07 Jul '22

These patches originate from my upcoming series bumping minimum qemu version to 4.2 and are valid even without the qemu version bump. Peter Krempa (11): test: domaincaps: Remove old test data qemuxml2(argv|xml)test: Drop qemu-3.1 version of 'cpu-Icelake-Server-pconfig' qemuxml2argvtest: Drop 'qemu-4.1' versions of hyperv tests qemuxml2argvtest: Convert 'cpu-eoi-(disabled|enabled)' cases to VIR_TEST_CAPS_LATEST qemuxml2argvtest: Remove qemu-4.0 versions of cpu feature test cases qemuxml2argvtest: Convert 'net-user' case to _LATEST qemuxml2argvtest: Make 'qemu-ns' test case more stable qemuxml2argvtest: Move real-caps versions of cpu-host tests out of the block setting fake host cpu qemu: Remove qemu-4.0 version of 'cpu-translation' test case qemucapabilitiestest: Fake proper version for 'caps_4.2.0.ppc64' case qemucapabilitiestest: Fake proper version for 'caps_4.2.0.aarch64' case .../domaincapsdata/qemu_2.11.0-q35.x86_64.xml | 191 --------------- .../domaincapsdata/qemu_2.11.0-tcg.x86_64.xml | 204 ---------------- tests/domaincapsdata/qemu_2.11.0.s390x.xml | 219 ----------------- tests/domaincapsdata/qemu_2.11.0.x86_64.xml | 191 --------------- .../domaincapsdata/qemu_2.12.0-q35.x86_64.xml | 212 ---------------- .../domaincapsdata/qemu_2.12.0-tcg.x86_64.xml | 222 ----------------- .../qemu_2.12.0-virt.aarch64.xml | 172 ------------- tests/domaincapsdata/qemu_2.12.0.aarch64.xml | 166 ------------- tests/domaincapsdata/qemu_2.12.0.ppc64.xml | 136 ----------- tests/domaincapsdata/qemu_2.12.0.s390x.xml | 219 ----------------- tests/domaincapsdata/qemu_2.12.0.x86_64.xml | 212 ---------------- .../domaincapsdata/qemu_3.0.0-q35.x86_64.xml | 208 ---------------- .../domaincapsdata/qemu_3.0.0-tcg.x86_64.xml | 220 ----------------- tests/domaincapsdata/qemu_3.0.0.ppc64.xml | 138 ----------- tests/domaincapsdata/qemu_3.0.0.s390x.xml | 226 ------------------ tests/domaincapsdata/qemu_3.0.0.x86_64.xml | 208 ---------------- .../caps_4.2.0.aarch64.replies | 4 +- .../caps_4.2.0.aarch64.xml | 2 +- .../caps_4.2.0.ppc64.replies | 4 +- .../qemucapabilitiesdata/caps_4.2.0.ppc64.xml | 2 +- ...u-Icelake-Server-pconfig.x86_64-3.1.0.args | 33 --- tests/qemuxml2argvdata/cpu-eoi-disabled.args | 31 --- ...gs => cpu-eoi-disabled.x86_64-latest.args} | 12 +- tests/qemuxml2argvdata/cpu-eoi-disabled.xml | 2 +- tests/qemuxml2argvdata/cpu-eoi-enabled.args | 31 --- ...rgs => cpu-eoi-enabled.x86_64-latest.args} | 13 +- tests/qemuxml2argvdata/cpu-eoi-enabled.xml | 2 +- .../cpu-translation.x86_64-4.0.0.args | 33 --- .../eoi-enabled.x86_64-4.0.0.args | 32 --- .../hyperv-panic.x86_64-4.0.0.args | 31 --- .../qemuxml2argvdata/hyperv.x86_64-4.0.0.args | 31 --- .../kvmclock+eoi-disabled.x86_64-4.0.0.args | 32 --- .../net-user.x86_64-4.0.0.args | 35 --- ...4.0.0.args => net-user.x86_64-latest.args} | 18 +- .../pv-spinlock-disabled.x86_64-4.0.0.args | 32 --- .../pv-spinlock-enabled.x86_64-4.0.0.args | 32 --- tests/qemuxml2argvdata/qemu-ns.xml | 1 - tests/qemuxml2argvtest.c | 24 +- ...pu-Icelake-Server-pconfig.x86_64-3.1.0.xml | 32 --- tests/qemuxml2xmloutdata/cpu-eoi-disabled.xml | 2 +- tests/qemuxml2xmloutdata/cpu-eoi-enabled.xml | 2 +- .../qemu-ns.x86_64-latest.xml | 1 - tests/qemuxml2xmltest.c | 1 - 43 files changed, 41 insertions(+), 3578 deletions(-) delete mode 100644 tests/domaincapsdata/qemu_2.11.0-q35.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_2.11.0-tcg.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_2.11.0.s390x.xml delete mode 100644 tests/domaincapsdata/qemu_2.11.0.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0-q35.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0-tcg.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0-virt.aarch64.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0.aarch64.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0.ppc64.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0.s390x.xml delete mode 100644 tests/domaincapsdata/qemu_2.12.0.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_3.0.0-q35.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_3.0.0-tcg.x86_64.xml delete mode 100644 tests/domaincapsdata/qemu_3.0.0.ppc64.xml delete mode 100644 tests/domaincapsdata/qemu_3.0.0.s390x.xml delete mode 100644 tests/domaincapsdata/qemu_3.0.0.x86_64.xml delete mode 100644 tests/qemuxml2argvdata/cpu-Icelake-Server-pconfig.x86_64-3.1.0.args delete mode 100644 tests/qemuxml2argvdata/cpu-eoi-disabled.args rename tests/qemuxml2argvdata/{hyperv-off.x86_64-4.0.0.args => cpu-eoi-disabled.x86_64-latest.args} (60%) delete mode 100644 tests/qemuxml2argvdata/cpu-eoi-enabled.args rename tests/qemuxml2argvdata/{eoi-disabled.x86_64-4.0.0.args => cpu-eoi-enabled.x86_64-latest.args} (60%) delete mode 100644 tests/qemuxml2argvdata/cpu-translation.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2argvdata/eoi-enabled.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2argvdata/hyperv-panic.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2argvdata/hyperv.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2argvdata/kvmclock+eoi-disabled.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2argvdata/net-user.x86_64-4.0.0.args rename tests/qemuxml2argvdata/{qemu-ns.x86_64-4.0.0.args => net-user.x86_64-latest.args} (61%) delete mode 100644 tests/qemuxml2argvdata/pv-spinlock-disabled.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2argvdata/pv-spinlock-enabled.x86_64-4.0.0.args delete mode 100644 tests/qemuxml2xmloutdata/cpu-Icelake-Server-pconfig.x86_64-3.1.0.xml -- 2.36.1

2 13

[PATCH 0/3] conf: Separate domain post parse code into domain_postparse.c
by Michal Privoznik 07 Jul '22

07 Jul '22

*** BLURB HERE *** Michal Prívozník (3): domain_conf: Unexport virDomainDefPostParseDeviceIteratorData conf: Separate domain post parse code into domain_postparse.c domain_postparse: Move error messages onto single line po/POTFILES | 1 + src/conf/domain_conf.c | 1446 +--------------------------------- src/conf/domain_conf.h | 10 - src/conf/domain_postparse.c | 1466 +++++++++++++++++++++++++++++++++++ src/conf/domain_postparse.h | 37 + src/conf/domain_validate.c | 11 +- src/conf/meson.build | 1 + src/libvirt_private.syms | 5 +- src/libxl/xen_xl.c | 1 + src/libxl/xen_xm.c | 1 + src/lxc/lxc_native.c | 1 + src/qemu/qemu_driver.c | 1 + src/qemu/qemu_process.c | 1 + src/vmx/vmx.c | 1 + 14 files changed, 1525 insertions(+), 1458 deletions(-) create mode 100644 src/conf/domain_postparse.c create mode 100644 src/conf/domain_postparse.h -- 2.35.1

2 4