[PATCH] qemuBackupDiskDataCleanupOne: Don't skip rest of cleanup if we can't enter monitor
by Peter Krempa
Recent fix to use the proper 'async' monitor function would cause
libvirt to leak some of the objects it's supposed to clean up in other
places besides qemu.
Don't skip the whole function on failure to enter the job but just the
monitor section.
Fixes: 9b22c25548aa658acdeac2269ddae32584df32d8
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
src/qemu/qemu_backup.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c
index 5eb2cbe306..f64639d501 100644
--- a/src/qemu/qemu_backup.c
+++ b/src/qemu/qemu_backup.c
@@ -125,10 +125,10 @@ qemuBackupDiskDataCleanupOne(virDomainObj *vm,
if (!dd->started) {
if (dd->added) {
- if (qemuDomainObjEnterMonitorAsync(vm, VIR_ASYNC_JOB_BACKUP) < 0)
- return;
- qemuBlockStorageSourceAttachRollback(priv->mon, dd->crdata->srcdata[0]);
- qemuDomainObjExitMonitor(vm);
+ if (qemuDomainObjEnterMonitorAsync(vm, VIR_ASYNC_JOB_BACKUP) == 0) {
+ qemuBlockStorageSourceAttachRollback(priv->mon, dd->crdata->srcdata[0]);
+ qemuDomainObjExitMonitor(vm);
+ }
}
if (dd->created) {
--
2.46.0
3 months, 2 weeks
[PATCH 0/3] hw/sh4: Remove the deprecated SHIX machine
by Philippe Mathieu-Daudé
Remove the deprecated SH4 SHIX machine, along
with the TC58128 NAND EEPROM.
Philippe Mathieu-Daudé (3):
hw/sh4: Remove the deprecated SHIX machine
hw/block: Remove TC58128 NAND EEPROM
hw/sh4: Remove sh7750_register_io_device() helper
MAINTAINERS | 11 --
docs/about/deprecated.rst | 6 -
docs/about/removed-features.rst | 5 +
configs/devices/sh4-softmmu/default.mak | 1 -
include/hw/sh4/sh.h | 19 ---
hw/block/tc58128.c | 211 ------------------------
hw/sh4/sh7750.c | 57 +------
hw/sh4/shix.c | 86 ----------
hw/block/Kconfig | 3 -
hw/block/meson.build | 1 -
hw/sh4/Kconfig | 7 -
hw/sh4/meson.build | 1 -
12 files changed, 7 insertions(+), 401 deletions(-)
delete mode 100644 hw/block/tc58128.c
delete mode 100644 hw/sh4/shix.c
--
2.45.2
3 months, 2 weeks
[libvirt PATCHv2 00/15] Switch to json-c from yajl
by Ján Tomko
Wow. Such patch. Much series.
Ján Tomko (15):
util: json: introduce virJSONStringPrettifyBlanks
tests: switch to compact empty JSON object formatting
build: introduce WITH_JSON
ci: install json-c too
meson: add option for building with json-c
meson: switch checks to depend on json-c as well as yajl
build: do not depend on yajl
build: link with json_c
util: json: write a json-c implementation
nss: convert findLeases to use json-c
nss: convert findMACs to use json-c
meson: do not link anything with yajl anymore
meson: options: drop yajl
meson: drop yajl detection
ci: drop yajl completely
ci/buildenv/almalinux-9.sh | 4 +-
ci/buildenv/alpine-319.sh | 4 +-
ci/buildenv/alpine-edge.sh | 4 +-
ci/buildenv/centos-stream-9.sh | 4 +-
ci/buildenv/debian-11-cross-aarch64.sh | 2 +-
ci/buildenv/debian-11-cross-armv6l.sh | 2 +-
ci/buildenv/debian-11-cross-armv7l.sh | 2 +-
ci/buildenv/debian-11-cross-i686.sh | 2 +-
ci/buildenv/debian-11-cross-mips64el.sh | 2 +-
ci/buildenv/debian-11-cross-mipsel.sh | 2 +-
ci/buildenv/debian-11-cross-ppc64le.sh | 2 +-
ci/buildenv/debian-11-cross-s390x.sh | 2 +-
ci/buildenv/debian-11.sh | 2 +-
ci/buildenv/debian-12-cross-aarch64.sh | 2 +-
ci/buildenv/debian-12-cross-armv6l.sh | 2 +-
ci/buildenv/debian-12-cross-armv7l.sh | 2 +-
ci/buildenv/debian-12-cross-i686.sh | 2 +-
ci/buildenv/debian-12-cross-mips64el.sh | 2 +-
ci/buildenv/debian-12-cross-mipsel.sh | 2 +-
ci/buildenv/debian-12-cross-ppc64le.sh | 2 +-
ci/buildenv/debian-12-cross-s390x.sh | 2 +-
ci/buildenv/debian-12.sh | 2 +-
ci/buildenv/debian-sid-cross-aarch64.sh | 2 +-
ci/buildenv/debian-sid-cross-armv6l.sh | 2 +-
ci/buildenv/debian-sid-cross-armv7l.sh | 2 +-
ci/buildenv/debian-sid-cross-i686.sh | 2 +-
ci/buildenv/debian-sid-cross-mips64el.sh | 2 +-
ci/buildenv/debian-sid-cross-ppc64le.sh | 2 +-
ci/buildenv/debian-sid-cross-s390x.sh | 2 +-
ci/buildenv/debian-sid.sh | 2 +-
ci/buildenv/fedora-39.sh | 4 +-
ci/buildenv/fedora-40.sh | 4 +-
ci/buildenv/fedora-rawhide.sh | 4 +-
ci/buildenv/opensuse-leap-15.sh | 2 +-
ci/buildenv/opensuse-tumbleweed.sh | 2 +-
ci/buildenv/ubuntu-2204.sh | 2 +-
ci/buildenv/ubuntu-2404.sh | 2 +-
ci/cirrus/freebsd-13.vars | 2 +-
ci/cirrus/freebsd-14.vars | 2 +-
ci/cirrus/macos-13.vars | 2 +-
ci/cirrus/macos-14.vars | 2 +-
ci/containers/almalinux-9.Dockerfile | 4 +-
ci/containers/alpine-319.Dockerfile | 4 +-
ci/containers/alpine-edge.Dockerfile | 4 +-
ci/containers/centos-stream-9.Dockerfile | 4 +-
.../debian-11-cross-aarch64.Dockerfile | 2 +-
.../debian-11-cross-armv6l.Dockerfile | 2 +-
.../debian-11-cross-armv7l.Dockerfile | 2 +-
ci/containers/debian-11-cross-i686.Dockerfile | 2 +-
.../debian-11-cross-mips64el.Dockerfile | 2 +-
.../debian-11-cross-mipsel.Dockerfile | 2 +-
.../debian-11-cross-ppc64le.Dockerfile | 2 +-
.../debian-11-cross-s390x.Dockerfile | 2 +-
ci/containers/debian-11.Dockerfile | 2 +-
.../debian-12-cross-aarch64.Dockerfile | 2 +-
.../debian-12-cross-armv6l.Dockerfile | 2 +-
.../debian-12-cross-armv7l.Dockerfile | 2 +-
ci/containers/debian-12-cross-i686.Dockerfile | 2 +-
.../debian-12-cross-mips64el.Dockerfile | 2 +-
.../debian-12-cross-mipsel.Dockerfile | 2 +-
.../debian-12-cross-ppc64le.Dockerfile | 2 +-
.../debian-12-cross-s390x.Dockerfile | 2 +-
ci/containers/debian-12.Dockerfile | 2 +-
.../debian-sid-cross-aarch64.Dockerfile | 2 +-
.../debian-sid-cross-armv6l.Dockerfile | 2 +-
.../debian-sid-cross-armv7l.Dockerfile | 2 +-
.../debian-sid-cross-i686.Dockerfile | 2 +-
.../debian-sid-cross-mips64el.Dockerfile | 2 +-
.../debian-sid-cross-ppc64le.Dockerfile | 2 +-
.../debian-sid-cross-s390x.Dockerfile | 2 +-
ci/containers/debian-sid.Dockerfile | 2 +-
ci/containers/fedora-39.Dockerfile | 4 +-
ci/containers/fedora-40.Dockerfile | 4 +-
ci/containers/fedora-rawhide.Dockerfile | 4 +-
ci/containers/opensuse-leap-15.Dockerfile | 2 +-
ci/containers/opensuse-tumbleweed.Dockerfile | 2 +-
ci/containers/ubuntu-2204.Dockerfile | 2 +-
ci/containers/ubuntu-2404.Dockerfile | 2 +-
ci/lcitool/projects/libvirt.yml | 2 +-
libvirt.spec.in | 6 +-
meson.build | 62 +--
meson_options.txt | 8 +-
src/libvirt_private.syms | 1 +
src/meson.build | 2 +-
src/util/meson.build | 2 +-
src/util/virjson.c | 485 +++++-------------
src/util/virjson.h | 2 +
tests/meson.build | 8 +-
tests/qemublocktest.c | 5 +-
.../backupmerge/empty-out.json | 4 +-
tests/qemumigparamsdata/empty.json | 4 +-
tests/qemumigparamstest.c | 5 +-
tests/virmacmaptest.c | 5 +-
tests/virmacmaptestdata/empty.json | 4 +-
tests/virnetdaemontest.c | 2 +-
tests/virstoragetest.c | 4 +-
tools/nss/libvirt_nss_leases.c | 370 +++++--------
tools/nss/libvirt_nss_macs.c | 278 +++-------
tools/nss/meson.build | 4 +-
99 files changed, 475 insertions(+), 972 deletions(-)
--
2.46.0
3 months, 2 weeks
[PATCH] qemu: backup: Use 'async' monitor in 'qemuBackupDiskDataCleanupOne'
by Peter Krempa
'qemuBackupDiskDataCleanupOne()' is entering the monitor while we're in
the async backup job inside 'qemuBackupBegin()' which is semantically
wrong and per upstream report causes crashes if some monitoring commands
are run in parallel.
Use qemuDomainObjEnterMonitorAsync() instead.
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
src/qemu/qemu_backup.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c
index 81391c29f7..5eb2cbe306 100644
--- a/src/qemu/qemu_backup.c
+++ b/src/qemu/qemu_backup.c
@@ -125,7 +125,8 @@ qemuBackupDiskDataCleanupOne(virDomainObj *vm,
if (!dd->started) {
if (dd->added) {
- qemuDomainObjEnterMonitor(vm);
+ if (qemuDomainObjEnterMonitorAsync(vm, VIR_ASYNC_JOB_BACKUP) < 0)
+ return;
qemuBlockStorageSourceAttachRollback(priv->mon, dd->crdata->srcdata[0]);
qemuDomainObjExitMonitor(vm);
}
--
2.46.0
3 months, 2 weeks
[PATCH 0/2] tests: qemucapabilities: Final update of 9.1 caps
by Peter Krempa
Update the qemu-9.1 caps on x86_64 to final version after yesterday's
qemu release and add caps for riscv64 captured on an x86_64 host to
manifest that ACPI is already supported:
https://gitlab.com/libvirt/libvirt/-/issues/665
Peter Krempa (2):
tests: qemucapabilities: Update 'caps_9.1.0_x86_64' for final release
tests: qemucapabilitiesdata: Add caps for 'caps_9.1.0_riscv64'
(captured on x86_64)
.../qemu_9.1.0-tcg-virt.riscv64.xml | 196 +
.../qemu_9.1.0-virt.riscv64.xml | 185 +
.../caps_9.1.0_riscv64.replies | 31586 ++++++++++++++++
.../caps_9.1.0_riscv64.xml | 193 +
.../caps_9.1.0_x86_64.replies | 14 +-
.../caps_9.1.0_x86_64.xml | 12 +-
...ult-video-type-riscv64.riscv64-latest.args | 3 +-
...ault-video-type-riscv64.riscv64-latest.xml | 3 +
...efi-riscv64.riscv64-latest.abi-update.args | 3 +-
...-efi-riscv64.riscv64-latest.abi-update.xml | 3 +
.../riscv64-virt-acpi.riscv64-latest.args | 3 +-
.../riscv64-virt-acpi.riscv64-latest.xml | 3 +
...ault-models.riscv64-latest.abi-update.args | 3 +-
...fault-models.riscv64-latest.abi-update.xml | 3 +
...64-virt-default-models.riscv64-latest.args | 3 +-
...v64-virt-default-models.riscv64-latest.xml | 3 +
.../riscv64-virt-graphics.riscv64-latest.args | 3 +-
.../riscv64-virt-graphics.riscv64-latest.xml | 3 +
...v64-virt-headless-mmio.riscv64-latest.args | 3 +-
...cv64-virt-headless-mmio.riscv64-latest.xml | 3 +
.../riscv64-virt-headless.riscv64-latest.args | 3 +-
.../riscv64-virt-headless.riscv64-latest.xml | 3 +
...irt-minimal.riscv64-latest.abi-update.args | 3 +-
...virt-minimal.riscv64-latest.abi-update.xml | 3 +
.../riscv64-virt-minimal.riscv64-latest.args | 3 +-
.../riscv64-virt-minimal.riscv64-latest.xml | 3 +
26 files changed, 32223 insertions(+), 23 deletions(-)
create mode 100644 tests/domaincapsdata/qemu_9.1.0-tcg-virt.riscv64.xml
create mode 100644 tests/domaincapsdata/qemu_9.1.0-virt.riscv64.xml
create mode 100644 tests/qemucapabilitiesdata/caps_9.1.0_riscv64.replies
create mode 100644 tests/qemucapabilitiesdata/caps_9.1.0_riscv64.xml
--
2.46.0
3 months, 2 weeks
[PATCH 0/3] virsh: Fix regression in script-friendly output of 'virsh list --uuid'
by Peter Krempa
Peter Krempa (3):
virsh: cmdList: Revert to script-friendly output for 'virsh list
--uuid'
NEWS: Mention breakage of scrip-friendly virsh output
virshtest: Add test case for script friendly options
NEWS.rst | 11 +++++++
tests/virshtest.c | 2 ++
.../virshtestdata/script-friendly-options.in | 11 +++++++
.../virshtestdata/script-friendly-options.out | 33 +++++++++++++++++++
tools/virsh-domain-monitor.c | 2 +-
5 files changed, 58 insertions(+), 1 deletion(-)
create mode 100644 tests/virshtestdata/script-friendly-options.in
create mode 100644 tests/virshtestdata/script-friendly-options.out
--
2.46.0
3 months, 2 weeks
Libvirtd sends SIGTERM to old qemu processes after restart
by Dmitrii Abramov
Hello, Libvirt community.
We have one strange issue with libivrtd.
We’ve been using Libvirtd in docker for several years. This year we switched to the new generation of processes AMD 7663 and we started to use the new version(for us) of libviirtd 8.0.0. Before this we used Libvirt 6.0
Right now we have such situation:
if we restart the container with Libvirt or Libvirt crashes and Docker engine restarts it the new process of Libvirt send SIGTERM to all running QEMU processes.
This highlight from strace of qemu process:
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=2045513, si_uid=0} ---
This is output from bash where you can see that PID 2045513 is a new process of Libvirt(look at the uptime of container and PID)
milan15 : ~ [0] # ps afxj | grep 2045513
2044884 2045700 2045699 2044884 pts/3 2045699 S+ 0 0:00 \_ grep --color=auto 2045513
2045492 2045513 2045513 2045513 ? -1 Ssl 0 0:04 \_ libvirtd -l
milan15 : ~ [0] #
milan15 : ~ [0] # docker ps | grep "libvirt$"
5b9e7d81a2f3 registry.beget.ru/vps/docker-libvirt/ubuntu/jammy:20240124-1 "libvirtd -l" 6 days ago Up 2 minutes libvirt
milan15 : ~ [0] # docker top libvirt
UID PID PPID C STIME TTY TIME CMD
root 2045513 2045492 2 21:20 ? 00:00:04 libvirtd -l
milan15 : ~ [0] #
We found that in logs Libvirtd says that he is unable to access to /sys/fs/cgroups.
{"log":"2024-09-04 17:40:02.831+0000: 2041803: error : virCgroupV2ParseControllersFile:282 : Unable to read from '/sys/fs/cgroup/../../machine/qemu-1394-mameluk-59ad0e58-732e-4468-9d3a-9be2cbac4931.libvirt-qemu/cgroup.controllers': No such file or directory\n","stream":"stderr","time":"2024-09-04T17:40:02.831308736Z"} {"log":"2024-09-04 17:40:02.831+0000: 2041804: error : virFileReadAll:1447 : Failed to open file '/sys/fs/cgroup/../../machine/qemu-1393-pavelaw3-215f218a-48d4-4b22-b15d-90ee0665f643.libvirt-qemu/cgroup.controllers': No such file or directory\n","stream":"stderr","time":"2024-09-04T17:40:02.83143703Z"} {"log":"2024-09-04 17:40:02.831+0000: 2041804: error : virCgroupV2ParseControllersFile:282 : Unable to read from ‘/sys/fs/cgroup/../../machine/qemu-1393-pavelaw3-215f218a-48d4-4b22-b15d-90ee0665f643.libvirt-qemu/cgroup.controllers': No such file or directory\n","stream":"stderr","time":"2024-09-04T17:40:02.831453382Z"}
We made several tests(added sleep and listed /sys/fs/cgroup directory before launching the new process ) — it seems that every process in container has ability to access to /sys/fs/cgroups.
It seems that the path /sys/fs/cgroup/../../machine/ isn’t correct.
We also tried Libvirt 10.0 version and many other. The result is the same.
We have the same scheme on other generations of processors — and everything works excellent.
This is an example of you docker-compose where we run Libvirt(just a part)
version: "3.0"
services:
libvirt:
image: mameluk_libvirtd
build:
context: .
privileged: true
volumes:
- /etc/docker/libvirt/etc/libvirt:/etc/libvirt:rw
- /lib/modules:/lib/modules:ro
- /dev:/dev
- /sys:/sys
- /run:/run:rw
- /var/lib/libvirt:/var/lib/libvirt:rslave
- /var/log/libvirt:/var/log/libvirt
- /home/docker/svc-libvirt/images:/home/svc-libvirt/images
- /etc/docker/libvirt/etc/lvm:/etc/lvm
- /home/docker/svc-libvirt/cidata:/home/svc-libvirt/cidata
ipc: host
network_mode: host
environment:
- TZ=Europe/Moscow
pid: host
restart: on-failure
entrypoint: ["/bin/start.sh", ""]
# depends_on:
#- virtlogd
container_name: mameluk_libvirtd
System: Ubuntu 22.04
core: Linux milan15 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Can anyone help us to solve this problem
--
Best Regards,
Dmitrii Abramov
3 months, 2 weeks
[RFC PATCH] deprecation: don't enable TCG plugins by default on 32 bit hosts
by Alex Bennée
The existing plugins already liberally use host pointer stuffing for
passing user data which will fail when doing 64 bit guests on 32 bit
hosts. We should discourage this by officially deprecating support and
adding another nail to the 32 bit host coffin.
Signed-off-by: Alex Bennée <alex.bennee(a)linaro.org>
---
docs/about/deprecated.rst | 11 +++++++++++
configure | 11 ++++++++++-
2 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index 88f0f03786..8a4e249717 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -184,6 +184,17 @@ be an effective use of its limited resources, and thus intends to discontinue
it. Since all recent x86 hardware from the past >10 years is capable of the
64-bit x86 extensions, a corresponding 64-bit OS should be used instead.
+TCG Plugin support not enabled by default on 32-bit hosts (since 9.2)
+'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+While it is still possible to enable TCG plugin support for 32-bit
+hosts there are a number of potential pitfalls when instrumenting
+64-bit guests. The plugin APIs typically pass most addresses as
+uint64_t but practices like encoding that address in a host pointer
+for passing as user-data will loose data. As most software analysis
+benefits from having plenty of host memory it seems reasonable to
+encourage users to use 64 bit builds of QEMU for analysis work
+whatever targets they are instrumenting.
System emulator CPUs
--------------------
diff --git a/configure b/configure
index d08b71f14b..8acb311527 100755
--- a/configure
+++ b/configure
@@ -424,6 +424,7 @@ fi
# Note that this case should only have supported host CPUs, not guests.
# Please keep it sorted and synchronized with meson.build's host_arch.
host_arch=
+host_bits=64
linux_arch=
case "$cpu" in
aarch64)
@@ -434,12 +435,14 @@ case "$cpu" in
armv*b|armv*l|arm)
cpu=arm
host_arch=arm
+ host_bits=32
linux_arch=arm
;;
i386|i486|i586|i686)
cpu="i386"
host_arch=i386
+ host_bits=32
linux_arch=x86
CPU_CFLAGS="-m32"
;;
@@ -458,11 +461,13 @@ case "$cpu" in
mips*)
cpu=mips
host_arch=mips
+ host_bits=32
linux_arch=mips
;;
ppc)
host_arch=ppc
+ host_bits=32
linux_arch=powerpc
CPU_CFLAGS="-m32"
;;
@@ -480,11 +485,13 @@ case "$cpu" in
riscv32 | riscv64)
host_arch=riscv
+ host_bits=32
linux_arch=riscv
;;
s390)
linux_arch=s390
+ host_bits=32
CPU_CFLAGS="-m31"
;;
s390x)
@@ -495,6 +502,7 @@ case "$cpu" in
sparc|sun4[cdmuv])
cpu=sparc
+ host_bits=32
CPU_CFLAGS="-m32 -mv8plus -mcpu=ultrasparc"
;;
sparc64)
@@ -505,6 +513,7 @@ case "$cpu" in
x32)
cpu="x86_64"
host_arch=x86_64
+ host_bits=32
linux_arch=x86
CPU_CFLAGS="-mx32"
;;
@@ -1028,7 +1037,7 @@ if test "$static" = "yes" ; then
fi
plugins="no"
fi
-if test "$plugins" != "no"; then
+if test "$plugins" != "no" && test host_bits = 64; then
plugins=yes
subdirs="$subdirs contrib/plugins"
fi
--
2.39.2
3 months, 2 weeks
[RFC PATCH] docs: add documentation for the AFL Fuzzing method
by Abhiram Tilak
I have worked on fuzzing libvirt with AFLPlusPlus over the summer and
documented my fuzzing setup [1], to help replicate my setup for anyone who
wants to take on fuzzing libvirt.
There are few ideas on how to move forward:
- Have the gitlab repo setup as it is, and if anyone is looking to
take forward fuzzing libvirt, they will find my repo in a
search-engine.
- Add a brief documented summary of my setup in the docs section
for libvirt repo. This would make it easier for anyone to find
my repo. The following patch adds the above mentioned summary
to the 'kbase/internals' section.
Adds a basic documentation required to replicate the fuzzing
setup and explains usage of various scripts.
The contents of this page is a ReStructured version of the
README found in the original setup repo [1].
[1]: https://gitlab.com/edupua/libvirt-aflfuzz
Signed-off-by: Abhiram Tilak <atp.exp(a)gmail.com>
---
docs/kbase/index.rst | 3 +
docs/kbase/internals/afl-fuzzing.rst | 275 +++++++++++++++++++++++++++
docs/kbase/internals/meson.build | 1 +
3 files changed, 279 insertions(+)
create mode 100644 docs/kbase/internals/afl-fuzzing.rst
diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst
index e51b35cbfc..1cbd2fc762 100644
--- a/docs/kbase/index.rst
+++ b/docs/kbase/index.rst
@@ -116,3 +116,6 @@ Internals
`QEMU monitor event handling <internals/qemu-event-handlers.html>`__
Brief outline how events emitted by qemu on the monitor are handlded.
+
+`XML Fuzzing using AFLPlusPlus <internals/afl-fuzzing.html>`__
+ Guide to use AFLPlusPlus for fuzzing libvirt's XML parsers.
diff --git a/docs/kbase/internals/afl-fuzzing.rst b/docs/kbase/internals/afl-fuzzing.rst
new file mode 100644
index 0000000000..2c37847476
--- /dev/null
+++ b/docs/kbase/internals/afl-fuzzing.rst
@@ -0,0 +1,275 @@
+============================
+AFL Fuzzing method
+============================
+
+Setup
+=====
+
+Basic Setup
+-----------
+
+Current setup uses a debian12 (bookworm) virtual machine. On a fresh
+installation, you will need the following packages installed on your
+system. Some really basic debian setup instructions are not mentioned
+here like adding user to sudo group etc.
+
+The helper scripts and other required files mentioned here can be found in
+[this](https://gitlab.com/edupua/libvirt-aflfuzz) repository. This
+repo will be referred as 'helper repo' in the following guide.
+
+- Libvirt Dependencies
+ (`source <https://gitlab.com/libvirt/libvirt/-/blob/master/ci/containers/debian-12....>`__)
+
+.. code:: sh
+
+ sudo apt install \
+ augeas-lenses augeas-tools bash-completion black ca-certificates \
+ ccache clang codespell cpp diffutils dnsmasq dwarves ebtables \
+ flake8 gcc gettext git grep iproute2 iptables kmod libacl1-dev \
+ libapparmor-dev libattr1-dev libaudit-dev libblkid-dev libc6-dev \
+ libcap-ng-dev libclang-rt-dev libcurl4-gnutls-dev libdevmapper-dev \
+ libfuse-dev libglib2.0-dev libglusterfs-dev libgnutls28-dev libiscsi-dev \
+ libnbd-dev libnl-3-dev libnl-route-3-dev libnuma-dev libparted-dev \
+ libpcap0.8-dev libpciaccess-dev librbd-dev libreadline-dev libsanlock-dev \
+ libsasl2-dev libselinux1-dev libssh-gcrypt-dev libssh2-1-dev libtirpc-dev \
+ libudev-dev libxen-dev libxml2-dev libxml2-utils libyajl-dev locales \
+ lvm2 make meson nfs-common ninja-build numad open-iscsi perl-base \
+ pkgconf policykit-1 python3 python3-docutils python3-pytest qemu-utils \
+ sed systemtap-sdt-dev wireshark-dev xsltproc apparmor
+
+- AFLPlusPlus Dependencies
+ (`source <https://github.com/AFLplusplus/AFLplusplus/blob/stable/docs/INSTALL.md>`__)
+
+.. code:: sh
+
+ sudo apt-get install \
+ build-essential python3-dev automake cmake git flex bison \
+ libglib2.0-dev libpixman-1-dev python3-setuptools cargo \
+ libgtk-3-dev lld llvm llvm-dev clang ninja-build
+
+ sudo apt-get install \
+ gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev \
+ libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev
+
+(separated the commands because the gcc-plugin-dev command requires gcc
+to be installed.)
+
+- By default the debian path doesn’t contain ``/sbin`` and
+ ``/usr/sbin``, its good to update the path to have them
+
+.. code:: sh
+
+ export PATH=/sbin:/usr/sbin/:/usr/local/sbin:$PATH
+
+Building AFLPlusPlus
+--------------------
+
+Build all the modules using the ``distrib`` target.
+
+.. code:: sh
+
+ git clone https://github.com/AFLplusplus/AFLplusplus
+ cd AFLplusplus
+ make PERFORMANCE=1 CODE_COVERAGE=1 PROFILING=1 INTROSPECTION=1 LLVM_CONFIG=llvm-config-14 distrib
+ sudo make install
+
+The following variables are the recommended one, but it is possible to have a
+more simpler build if one wants to.
+
+Bulding AFL-Cov
+---------------
+
+.. code:: sh
+
+ git clone https://github.com/vanhauser-thc/afl-cov.git
+ cd afl-cov
+ sudo make install
+
+Install tmux
+------------
+
+.. code:: sh
+
+ sudo apt install tmux
+
+Building Libvirt with AFL harness
+=================================
+
+Currently there is a fork of
+`libvirt <https://gitlab.com/edupua/libvirt>`__ which adds support for
+compiling the required harnesses.
+
+[Note: This repo might not be upto date with upstream, please rebase]
+
+The directory structure is really important because all the helper
+scripts rely on it.
+
+- Libvirt is expected to be cloned in the home directory.
+
+ .. code:: sh
+
+ cd ~/libvirt
+ git clone https://gitlab.com/libvirt/libvirt
+
+Alternatively there is a patch series with required commits. You can
+find them in the ``patches`` directory in the helper repo. Again, this might
+not be upto date with the current repo and might need conflict handling.
+
+Apply all the patches in order, except the cover-letter.
+
+.. code:: sh
+
+ git am patches/0001-afl-fuzzing-add-meson-support-for-fuzzing.patch
+ git am patches/0002-afl-fuzzzing-add-harness-for-net-create-and-net-port.patch
+ git am patches/0003-afl-fuzzzing-add-harness-for-nwfilter.patch
+ git am patches/0004-afl-fuzzzing-add-harness-for-vol-create-and-pool-cre.patch
+ git am patches/0005-afl-fuzzzing-add-harness-for-domain-create.patch
+
+Build Script
+------------
+
+The first script to be used is the build script which can be found in the
+helper repo. It can build individual builds with support for different
+compilers and different sanitizers. It also has a cmplog build which
+requires a corresponding default build to run. (Using cmplog requires
+2 repos, the cmplog build and a non-cmplog AFL build).
+
+.. code:: sh
+
+ scripts/build.sh --all
+
+The full build is about 15 Gigs, please make sure you have provision for
+it... code:: sh
+
+ cd ~/libvirt
+ git clone https://gitlab.com/libvirt/libvirt
+
+The default build contains ``undefined sanitizer`` and
+``address sanitizer``, there are other options like ``tsan`` and
+``AFL_HARDEN`` are also produced.
+
+Corpus generation
+=================
+
+There is a python generator tool
+```lxmlgen`` <https://gitlab.com/edupua/lxmlgen>`__ which can be used to
+generate a decent corpus useful for fuzzing most commands. The required
+documentation is in the repo itself and consists of just running two
+scripts ``run.sh`` and ``cmin.sh``.
+
+.. code:: sh
+
+ git clone https://gitlab.com/edupua/lxmlgen
+ cd lxmlgen
+ ./run.sh
+ ./cmin.sh --all
+
+The cmin.sh expects the default build to exist (``build_default``),
+which should be generated in the previous step.
+
+The fuzzing scripts expect the final corpus to be in the ``corpus``
+directory in the root of the current repo.
+
+Fuzzing Individual builds
+=========================
+
+Up from this point, a casual fuzzer can start fuzzing runs. Use the
+helper scripts ``base.sh`` or ``custom_mutator.sh`` that pass extra
+arguments to ``afl-fuzz`` thus, there are
+`plethora <https://www.mankier.com/8/afl-fuzz>`__ of options to play
+around with.
+
+.. code:: sh
+
+ scripts/base.sh net-create build_default
+
+Custom Mutator
+--------------
+
+There is a simple `custom mutator <https://gitlab.com/edupua/mutator>`__
+write in python. To take full advantage of this mutator, please use it
+singly with ``AFL_CUSTOM_MUTATOR_ONLY`` option.
+
+Once you clone the repo don’t forget to install the ``requirements.txt``
+and you can use the script ``custom_mutator.sh`` just like ``base.sh``.
+
+This script expects the mutator to be cloned into a folder called
+``custom_mutator``, with the python libraries installed. It also expects
+the corresponding RelaxNG schema files to be placed in ``schemas`` dir,
+which should be present by default.
+
+Generate a tmux command
+=======================
+
+If you have access to a large amount of computational power with
+multiple cores it is recommended to run a parallel fuzz with multiple
+instances running at once. Its meant to only run one afl subcommand at
+once.
+
+The script ``gen_tmux.sh`` automatically detects all the build
+directories in the libvirt directory and uses a relevant combination of
+``afl-fuzz`` options with different weights for each option.
+
+It also has supports for ``--custom_mutator`` argument which uses
+``custom_mutator.sh`` instead of ``base.sh``.
+
+.. code:: sh
+
+ scripts/gen_tmux.sh net-create --custom_mutator
+
+Finally this script should generate a ``mega.sh`` which deploys multiple
+instances. The script generates approximately 30 instances and each run
+uses a core of CPU, and may require manually adding or removing
+instances accordingly.
+
+Gather Coverage
+===============
+
+There is again a script to generate coverage for a given subcommand.
+
+.. code:: sh
+
+ scripts/cov.sh net-create
+
+Note: This command can take a really long time since it generates
+coverage for each and every queue file in the corpus and does so for all
+the master plus slave instances of a fuzzing run.
+
+You can use the argument ``--cover-corpus`` which avoids running each
+file and is lot faster.
+
+The final coverage can be viewed in HTML format in the ``cov/web``
+directory inside the specified output folder.
+
+See Also
+========
+
+Setting up AFLPlusPlus in a new machine may require a tuning. Please use
+the following resources:
+
+- Ultimate fuzzing using AFLPlusPlus guide -
+ https://aflplus.plus/docs/fuzzing_in_depth/
+- Performance Tips in AFLPlusPlus -
+ https://aflplus.plus/docs/perf_tips/
+- Parallel fuzzing - https://aflplus.plus/docs/parallel_fuzzing/
+
+There is also detailed documentation that can be easily found in AFL’s
+github repo: - `Custom
+mutators <https://github.com/AFLplusplus/AFLplusplus/tree/stable/custom_mutators>`__
+- Use this guide to help craft custom mutators for AFLPlusPlus, also
+take a look at the reference links to find some related links. -
+`Instrumentation <https://github.com/AFLplusplus/AFLplusplus/tree/stable/instrumentation>`__
+- This directory contains a detailed list of popular instrumentation
+methods used and some related examples.
+
+It is also really important to have a great understand of how AFL works
+under the hood before using it to do something useful. Refer to `this
+blog <https://blog.ritsec.club/posts/afl-under-hood/>`__
+
+Please consider looking at other efforts to fuzz libvirt as well:
+
+- AltLinux: `Fuzzing libvirt
+ RPC <https://www.altlinux.org/Libvirt_fuzzing_technique>`__
+- Libfuzzer + Libprotobuf-mutator: `Fuzzing libvirt using grammar aware
+ mutator <https://gitlab.com/Skryptonyte/libvirt/-/tree/gsoc_fuzz_final_forreal/tes...>`__
+
diff --git a/docs/kbase/internals/meson.build b/docs/kbase/internals/meson.build
index f1e9122f8f..b944c50e0b 100644
--- a/docs/kbase/internals/meson.build
+++ b/docs/kbase/internals/meson.build
@@ -9,6 +9,7 @@ docs_kbase_internals_files = [
'qemu-migration',
'qemu-threads',
'rpc',
+ 'afl-fuzzing',
]
--
2.39.2
3 months, 2 weeks
[PATCH 00/14] Introduce fuzzing of XML formats
by Rayhan Faizel
This series introduces multiple fuzzers developed as part of Google Summer
of Code 2024. We adopt a structure-aware fuzzing approach to fuzz libvirt
XML formats. The fuzzing methodology makes use of libFuzzer and
libprotobuf-mutator. The fuzzers work by mutating intermediate protobufs
and converting them to XML.
The fuzzing method in use requires inclusion of C++ sources. However, C++
compilation will be done only if '-Dfuzz' is enabled. Otherwise, libvirt will
compile normally as before. The fuzzing method works only on clang compilers
which support libFuzzer.
This series introduces a total of six fuzzers:
1. QEMU XML domain
2. QEMU XML hotplug
3. CH XML domain
4. VMX XML domain
5. libXL XML domain
6. NWFilter XML
In terms of the number of crashes discovered, QEMU XML domain, QEMU XML
hotplug and libXL fuzzers are the most interesting ones.
The setup process is documented at the end of the series (patch 14).
Rayhan Faizel (14):
src: Tweak source code to allow C++ compilation
meson: Add support for clang/LLVM coverage instrumentation
tests: Export handlers for fake secondary drivers
schemas: Refactor relaxNG schema to ease protobuf conversion
scripts: Add script to convert relaxNG to protobuf
fuzz: Implement base fuzzing setup for XML domain
fuzz: Implement QEMU XML domain fuzzer
fuzz: Implement QEMU XML hotplug fuzzer
ch: Remove unused variables
fuzz: Implement CH XML domain fuzzer
fuzz: Implement VMX XML domain fuzzer
fuzz: Implement libXL XML domain fuzzer
fuzz: Implement NWFilter XML fuzzer
docs: Document the fuzzers
build-aux/syntax-check.mk | 1 +
docs/kbase/index.rst | 3 +
docs/kbase/internals/meson.build | 1 +
docs/kbase/internals/xml-fuzzing.rst | 120 ++++
meson.build | 55 ++
meson_options.txt | 5 +-
scripts/meson.build | 1 +
scripts/relaxng-to-proto.py | 521 ++++++++++++++++++
src/ch/ch_monitor.c | 2 +-
src/ch/ch_monitor.h | 3 +
src/ch/ch_process.c | 2 -
src/conf/domain_conf.c | 18 +-
src/conf/domain_conf.h | 6 +-
src/conf/netdev_vport_profile_conf.c | 2 +-
src/conf/schemas/basictypes.rng | 20 +-
src/conf/schemas/domaincommon.rng | 11 +-
src/conf/schemas/networkcommon.rng | 14 +-
src/qemu/qemu_hotplug.c | 4 +
src/qemu/qemu_monitor.c | 6 +-
src/qemu/qemu_monitor.h | 2 +-
src/util/virfile.h | 2 +-
src/util/virnetdev.h | 12 +-
src/util/virnetdevip.h | 2 +-
src/util/virnetdevmacvlan.h | 2 +-
src/util/virnetdevvportprofile.c | 2 +-
src/util/virnetdevvportprofile.h | 2 +-
src/util/virnvme.c | 4 +-
src/util/virnvme.h | 2 +-
src/util/viruuid.h | 2 +-
tests/commandhelper.c | 8 +-
tests/fuzz/README.rst | 131 +++++
tests/fuzz/ch_xml_domain_fuzz.cc | 157 ++++++
tests/fuzz/libxl_xml_domain_fuzz.cc | 159 ++++++
tests/fuzz/llvm_symbolizer_wrapper.c | 11 +
tests/fuzz/meson.build | 183 ++++++
tests/fuzz/proto_custom_datatypes.cc | 234 ++++++++
tests/fuzz/proto_custom_datatypes.h | 30 +
tests/fuzz/proto_header_common.h | 51 ++
tests/fuzz/proto_to_xml.cc | 277 ++++++++++
tests/fuzz/proto_to_xml.h | 39 ++
tests/fuzz/protos/meson.build | 46 ++
tests/fuzz/protos/xml_datatypes.proto | 93 ++++
tests/fuzz/protos/xml_domain.proto | 62 +++
tests/fuzz/protos/xml_domain_disk_only.proto | 21 +
.../protos/xml_domain_interface_only.proto | 21 +
tests/fuzz/protos/xml_hotplug.proto | 38 ++
tests/fuzz/protos/xml_nwfilter.proto | 9 +
tests/fuzz/qemu_xml_domain_fuzz.cc | 277 ++++++++++
tests/fuzz/qemu_xml_hotplug_fuzz.cc | 340 ++++++++++++
tests/fuzz/run_fuzz.in | 142 +++++
tests/fuzz/vmx_xml_domain_fuzz.cc | 208 +++++++
tests/fuzz/xml_nwfilter_fuzz.cc | 149 +++++
tests/meson.build | 5 +
tests/qemumonitortestutils.c | 48 ++
tests/qemumonitortestutils.h | 6 +
tests/qemuxmlconftest.c | 249 ---------
tests/testutilsqemu.c | 256 +++++++++
tests/testutilsqemu.h | 57 ++
58 files changed, 3832 insertions(+), 302 deletions(-)
create mode 100644 docs/kbase/internals/xml-fuzzing.rst
create mode 100644 scripts/relaxng-to-proto.py
create mode 100644 tests/fuzz/README.rst
create mode 100644 tests/fuzz/ch_xml_domain_fuzz.cc
create mode 100644 tests/fuzz/libxl_xml_domain_fuzz.cc
create mode 100644 tests/fuzz/llvm_symbolizer_wrapper.c
create mode 100644 tests/fuzz/meson.build
create mode 100644 tests/fuzz/proto_custom_datatypes.cc
create mode 100644 tests/fuzz/proto_custom_datatypes.h
create mode 100644 tests/fuzz/proto_header_common.h
create mode 100644 tests/fuzz/proto_to_xml.cc
create mode 100644 tests/fuzz/proto_to_xml.h
create mode 100644 tests/fuzz/protos/meson.build
create mode 100644 tests/fuzz/protos/xml_datatypes.proto
create mode 100644 tests/fuzz/protos/xml_domain.proto
create mode 100644 tests/fuzz/protos/xml_domain_disk_only.proto
create mode 100644 tests/fuzz/protos/xml_domain_interface_only.proto
create mode 100644 tests/fuzz/protos/xml_hotplug.proto
create mode 100644 tests/fuzz/protos/xml_nwfilter.proto
create mode 100644 tests/fuzz/qemu_xml_domain_fuzz.cc
create mode 100644 tests/fuzz/qemu_xml_hotplug_fuzz.cc
create mode 100644 tests/fuzz/run_fuzz.in
create mode 100644 tests/fuzz/vmx_xml_domain_fuzz.cc
create mode 100644 tests/fuzz/xml_nwfilter_fuzz.cc
--
2.34.1
3 months, 2 weeks
