I have worked on fuzzing libvirt with AFLPlusPlus over the summer and documented my fuzzing setup [1], to help replicate my setup for anyone who wants to take on fuzzing libvirt. There are few ideas on how to move forward: - Have the gitlab repo setup as it is, and if anyone is looking to take forward fuzzing libvirt, they will find my repo in a search-engine. - Add a brief documented summary of my setup in the docs section for libvirt repo. This would make it easier for anyone to find my repo. The following patch adds the above mentioned summary to the 'kbase/internals' section. Adds a basic documentation required to replicate the fuzzing setup and explains usage of various scripts. The contents of this page is a ReStructured version of the README found in the original setup repo [1]. [1]: https://gitlab.com/edupua/libvirt-aflfuzz Signed-off-by: Abhiram Tilak <atp.exp@gmail.com> --- docs/kbase/index.rst | 3 + docs/kbase/internals/afl-fuzzing.rst | 275 +++++++++++++++++++++++++++ docs/kbase/internals/meson.build | 1 + 3 files changed, 279 insertions(+) create mode 100644 docs/kbase/internals/afl-fuzzing.rst diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index e51b35cbfc..1cbd2fc762 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -116,3 +116,6 @@ Internals `QEMU monitor event handling <internals/qemu-event-handlers.html>`__ Brief outline how events emitted by qemu on the monitor are handlded. + +`XML Fuzzing using AFLPlusPlus <internals/afl-fuzzing.html>`__ + Guide to use AFLPlusPlus for fuzzing libvirt's XML parsers. diff --git a/docs/kbase/internals/afl-fuzzing.rst b/docs/kbase/internals/afl-fuzzing.rst new file mode 100644 index 0000000000..2c37847476 --- /dev/null +++ b/docs/kbase/internals/afl-fuzzing.rst @@ -0,0 +1,275 @@ +============================ +AFL Fuzzing method +============================ + +Setup +===== + +Basic Setup +----------- + +Current setup uses a debian12 (bookworm) virtual machine. On a fresh +installation, you will need the following packages installed on your +system. Some really basic debian setup instructions are not mentioned +here like adding user to sudo group etc. + +The helper scripts and other required files mentioned here can be found in +[this](https://gitlab.com/edupua/libvirt-aflfuzz) repository. This +repo will be referred as 'helper repo' in the following guide. + +- Libvirt Dependencies + (`source <https://gitlab.com/libvirt/libvirt/-/blob/master/ci/containers/debian-12.Dockerfile>`__) + +.. code:: sh + + sudo apt install \ + augeas-lenses augeas-tools bash-completion black ca-certificates \ + ccache clang codespell cpp diffutils dnsmasq dwarves ebtables \ + flake8 gcc gettext git grep iproute2 iptables kmod libacl1-dev \ + libapparmor-dev libattr1-dev libaudit-dev libblkid-dev libc6-dev \ + libcap-ng-dev libclang-rt-dev libcurl4-gnutls-dev libdevmapper-dev \ + libfuse-dev libglib2.0-dev libglusterfs-dev libgnutls28-dev libiscsi-dev \ + libnbd-dev libnl-3-dev libnl-route-3-dev libnuma-dev libparted-dev \ + libpcap0.8-dev libpciaccess-dev librbd-dev libreadline-dev libsanlock-dev \ + libsasl2-dev libselinux1-dev libssh-gcrypt-dev libssh2-1-dev libtirpc-dev \ + libudev-dev libxen-dev libxml2-dev libxml2-utils libyajl-dev locales \ + lvm2 make meson nfs-common ninja-build numad open-iscsi perl-base \ + pkgconf policykit-1 python3 python3-docutils python3-pytest qemu-utils \ + sed systemtap-sdt-dev wireshark-dev xsltproc apparmor + +- AFLPlusPlus Dependencies + (`source <https://github.com/AFLplusplus/AFLplusplus/blob/stable/docs/INSTALL.md>`__) + +.. code:: sh + + sudo apt-get install \ + build-essential python3-dev automake cmake git flex bison \ + libglib2.0-dev libpixman-1-dev python3-setuptools cargo \ + libgtk-3-dev lld llvm llvm-dev clang ninja-build + + sudo apt-get install \ + gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev \ + libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev + +(separated the commands because the gcc-plugin-dev command requires gcc +to be installed.) + +- By default the debian path doesn’t contain ``/sbin`` and + ``/usr/sbin``, its good to update the path to have them + +.. code:: sh + + export PATH=/sbin:/usr/sbin/:/usr/local/sbin:$PATH + +Building AFLPlusPlus +-------------------- + +Build all the modules using the ``distrib`` target. + +.. code:: sh + + git clone https://github.com/AFLplusplus/AFLplusplus + cd AFLplusplus + make PERFORMANCE=1 CODE_COVERAGE=1 PROFILING=1 INTROSPECTION=1 LLVM_CONFIG=llvm-config-14 distrib + sudo make install + +The following variables are the recommended one, but it is possible to have a +more simpler build if one wants to. + +Bulding AFL-Cov +--------------- + +.. code:: sh + + git clone https://github.com/vanhauser-thc/afl-cov.git + cd afl-cov + sudo make install + +Install tmux +------------ + +.. code:: sh + + sudo apt install tmux + +Building Libvirt with AFL harness +================================= + +Currently there is a fork of +`libvirt <https://gitlab.com/edupua/libvirt>`__ which adds support for +compiling the required harnesses. + +[Note: This repo might not be upto date with upstream, please rebase] + +The directory structure is really important because all the helper +scripts rely on it. + +- Libvirt is expected to be cloned in the home directory. + + .. code:: sh + + cd ~/libvirt + git clone https://gitlab.com/libvirt/libvirt + +Alternatively there is a patch series with required commits. You can +find them in the ``patches`` directory in the helper repo. Again, this might +not be upto date with the current repo and might need conflict handling. + +Apply all the patches in order, except the cover-letter. + +.. code:: sh + + git am patches/0001-afl-fuzzing-add-meson-support-for-fuzzing.patch + git am patches/0002-afl-fuzzzing-add-harness-for-net-create-and-net-port.patch + git am patches/0003-afl-fuzzzing-add-harness-for-nwfilter.patch + git am patches/0004-afl-fuzzzing-add-harness-for-vol-create-and-pool-cre.patch + git am patches/0005-afl-fuzzzing-add-harness-for-domain-create.patch + +Build Script +------------ + +The first script to be used is the build script which can be found in the +helper repo. It can build individual builds with support for different +compilers and different sanitizers. It also has a cmplog build which +requires a corresponding default build to run. (Using cmplog requires +2 repos, the cmplog build and a non-cmplog AFL build). + +.. code:: sh + + scripts/build.sh --all + +The full build is about 15 Gigs, please make sure you have provision for +it... code:: sh + + cd ~/libvirt + git clone https://gitlab.com/libvirt/libvirt + +The default build contains ``undefined sanitizer`` and +``address sanitizer``, there are other options like ``tsan`` and +``AFL_HARDEN`` are also produced. + +Corpus generation +================= + +There is a python generator tool +```lxmlgen`` <https://gitlab.com/edupua/lxmlgen>`__ which can be used to +generate a decent corpus useful for fuzzing most commands. The required +documentation is in the repo itself and consists of just running two +scripts ``run.sh`` and ``cmin.sh``. + +.. code:: sh + + git clone https://gitlab.com/edupua/lxmlgen + cd lxmlgen + ./run.sh + ./cmin.sh --all + +The cmin.sh expects the default build to exist (``build_default``), +which should be generated in the previous step. + +The fuzzing scripts expect the final corpus to be in the ``corpus`` +directory in the root of the current repo. + +Fuzzing Individual builds +========================= + +Up from this point, a casual fuzzer can start fuzzing runs. Use the +helper scripts ``base.sh`` or ``custom_mutator.sh`` that pass extra +arguments to ``afl-fuzz`` thus, there are +`plethora <https://www.mankier.com/8/afl-fuzz>`__ of options to play +around with. + +.. code:: sh + + scripts/base.sh net-create build_default + +Custom Mutator +-------------- + +There is a simple `custom mutator <https://gitlab.com/edupua/mutator>`__ +write in python. To take full advantage of this mutator, please use it +singly with ``AFL_CUSTOM_MUTATOR_ONLY`` option. + +Once you clone the repo don’t forget to install the ``requirements.txt`` +and you can use the script ``custom_mutator.sh`` just like ``base.sh``. + +This script expects the mutator to be cloned into a folder called +``custom_mutator``, with the python libraries installed. It also expects +the corresponding RelaxNG schema files to be placed in ``schemas`` dir, +which should be present by default. + +Generate a tmux command +======================= + +If you have access to a large amount of computational power with +multiple cores it is recommended to run a parallel fuzz with multiple +instances running at once. Its meant to only run one afl subcommand at +once. + +The script ``gen_tmux.sh`` automatically detects all the build +directories in the libvirt directory and uses a relevant combination of +``afl-fuzz`` options with different weights for each option. + +It also has supports for ``--custom_mutator`` argument which uses +``custom_mutator.sh`` instead of ``base.sh``. + +.. code:: sh + + scripts/gen_tmux.sh net-create --custom_mutator + +Finally this script should generate a ``mega.sh`` which deploys multiple +instances. The script generates approximately 30 instances and each run +uses a core of CPU, and may require manually adding or removing +instances accordingly. + +Gather Coverage +=============== + +There is again a script to generate coverage for a given subcommand. + +.. code:: sh + + scripts/cov.sh net-create + +Note: This command can take a really long time since it generates +coverage for each and every queue file in the corpus and does so for all +the master plus slave instances of a fuzzing run. + +You can use the argument ``--cover-corpus`` which avoids running each +file and is lot faster. + +The final coverage can be viewed in HTML format in the ``cov/web`` +directory inside the specified output folder. + +See Also +======== + +Setting up AFLPlusPlus in a new machine may require a tuning. Please use +the following resources: + +- Ultimate fuzzing using AFLPlusPlus guide - + https://aflplus.plus/docs/fuzzing_in_depth/ +- Performance Tips in AFLPlusPlus - + https://aflplus.plus/docs/perf_tips/ +- Parallel fuzzing - https://aflplus.plus/docs/parallel_fuzzing/ + +There is also detailed documentation that can be easily found in AFL’s +github repo: - `Custom +mutators <https://github.com/AFLplusplus/AFLplusplus/tree/stable/custom_mutators>`__ +- Use this guide to help craft custom mutators for AFLPlusPlus, also +take a look at the reference links to find some related links. - +`Instrumentation <https://github.com/AFLplusplus/AFLplusplus/tree/stable/instrumentation>`__ +- This directory contains a detailed list of popular instrumentation +methods used and some related examples. + +It is also really important to have a great understand of how AFL works +under the hood before using it to do something useful. Refer to `this +blog <https://blog.ritsec.club/posts/afl-under-hood/>`__ + +Please consider looking at other efforts to fuzz libvirt as well: + +- AltLinux: `Fuzzing libvirt + RPC <https://www.altlinux.org/Libvirt_fuzzing_technique>`__ +- Libfuzzer + Libprotobuf-mutator: `Fuzzing libvirt using grammar aware + mutator <https://gitlab.com/Skryptonyte/libvirt/-/tree/gsoc_fuzz_final_forreal/tests/fuzz?ref_type=heads>`__ + diff --git a/docs/kbase/internals/meson.build b/docs/kbase/internals/meson.build index f1e9122f8f..b944c50e0b 100644 --- a/docs/kbase/internals/meson.build +++ b/docs/kbase/internals/meson.build @@ -9,6 +9,7 @@ docs_kbase_internals_files = [ 'qemu-migration', 'qemu-threads', 'rpc', + 'afl-fuzzing', ] -- 2.39.2