Enabling core dumps is a reasonably straightforward task, but is not documented clearly. This page provides as easy link to point users to when they need to debug QEMU. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- docs/kbase/index.rst | 4 ++ docs/kbase/meson.build | 1 + docs/kbase/qemu-core-dump.rst | 132 ++++++++++++++++++++++++++++++++++ 3 files changed, 137 insertions(+) create mode 100644 docs/kbase/qemu-core-dump.rst diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index 91083ee49d..372042886d 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -67,3 +67,7 @@ Internals / Debugging `VM migration internals <migrationinternals.html>`__ VM migration implementation details, complementing the info in `migration <../migration.html>`__ + +`Capturing core dumps for QEMU <qemu-core-dump.html>`__ + How to configure libvirt to enable capture of core dumps from + QEMU virtual machines diff --git a/docs/kbase/meson.build b/docs/kbase/meson.build index 7631b47018..6d17a83d1d 100644 --- a/docs/kbase/meson.build +++ b/docs/kbase/meson.build @@ -12,6 +12,7 @@ docs_kbase_files = [ 'locking-sanlock', 'merging_disk_image_chains', 'migrationinternals', + 'qemu-core-dump', 'qemu-passthrough-security', 'rpm-deployment', 's390_protected_virt', diff --git a/docs/kbase/qemu-core-dump.rst b/docs/kbase/qemu-core-dump.rst new file mode 100644 index 0000000000..d27f81c4d6 --- /dev/null +++ b/docs/kbase/qemu-core-dump.rst @@ -0,0 +1,132 @@ +============================= +Capturing core dumps for QEMU +============================= + +The default behaviour for a QEMU virtual machine launched by libvirt is to +have core dumps disabled. There can be times, however, when it is beneficial +to collect a core dump to enable debugging. + +QEMU driver configuration +========================= + +There is a global setting in the QEMU driver configuration file that controls +whether core dumps are permitted, and their maximum size. Enabling core dumps +is simply a matter of setting the maximum size to a non-zero value by editting +the ``/etc/libvirt/qemu.conf`` file: + +:: + + max_core = "unlimited" + +For an adhoc debugging session, setting the core dump size to "unlimited" is +viable, on the assumption that the core dumps will be disabled again once the +requisite information is collected. If the intention is to leave core dumps +permanently enabled, more careful consideration of limits is required + +Note that by default, a core dump will **NOT** include the the guest RAM +region, so will only include memory regions used by QEMU for emulation and +backend purposes. This is expected to be sufficient for the vast majority +of debugging needs. + +When there is a need to examine guest RAM though, a further setting is +available + +:: + + dump_guest_core = 1 + +This will of course result in core dumps that are as large as the biggest +virtual machine on the host - potentially 10's or even 100's of GB in size. +To allow more fine grained control it is possible to toggle this on a per +VM basis in the XML configuration. + +After changing either of the settings in ``/etc/libvirt/qemu.conf`` the daemon +hosting the QEMU driver must be restarted. For deployments using the monolithic +daemons, this means ``libvirtd``, while for those using modular daemons this +means ``virtqemud`` + +:: + + systemctl restart libvirtd (for a monolithic deployment) + systemctl restart virtqemud (for a modular deployment) + +While libvirt attempts to make it possible to restart the daemons without +negatively impacting running guests, there are some management operations +that may get interrupted. In particular long running jobs like live +migration or block device copy jobs may abort. It is thus wise to check +that the host is mostly idle before restarting the daemons. + +Guest core dump configuration +============================= + +The ``dump_guest_core`` setting mentioned above will allow guest RAM to be +included in core dumps for all virtual machines on the host. This may not +be desirable, so it is also possible to control this on a per-virtual +machine basis in the XML configuration: + +:: + + <memory dumpCore="on">...</memory> + +Note, it is still neccessary to at least set ``max_core`` to a non-zero +value in the global configuration file. + +Some management applications may not offer the ability to customimze the +XML configuration for a guest. In such situations, using the global +``dump_guest_core`` setting is the only option. + +Host OS core dump storage +========================= + +The Linux kernel default behaviour is to write core dumps to a file in the +current working directory of the process. This will not work with QEMU +processes launched by libvirt, because their working directory is ``/`` +which will not be writable. + +Most modern OS distros, however, now include systemd which configures a +custom core dump handler out of the box. When this is in effect, core dumps +from QEMU can be seen using the ``coredumpctl`` commands + +:: + + $ coredumpctl list -r + TIME PID UID GID SIG COREFILE EXE SIZE + Tue 2021-07-20 12:12:52 BST 2649303 107 107 SIGABRT present /usr/bin/qemu-system-x86_64 1.8M + ...snip... + + $ coredumpctl info 2649303 + PID: 2649303 (qemu-system-x86) + UID: 107 (qemu) + GID: 107 (qemu) + Signal: 6 (ABRT) + Timestamp: Tue 2021-07-20 12:12:52 BST (48min ago) + Command Line: /usr/bin/qemu-system-x86_64 -name guest=f30,debug-threads=on ..snip... -msg timestamp=on + Executable: /usr/bin/qemu-system-x86_64 + Control Group: /machine.slice/machine-qemu\x2d1\x2df30.scope/libvirt/emulator + Unit: machine-qemu\x2d1\x2df30.scope + Slice: machine.slice + Boot ID: 6b9015d0c05f4e7fbfe4197a2c7824a2 + Machine ID: c78c8286d6d74b22ac0dd275975f9ced + Hostname: localhost.localdomain + Storage: /var/lib/systemd/coredump/core.qemu-system-x86.107.6b9015d0c05f4e7fbfe4197a2c7824a2.2649303.1626779572000000.zst (present) + Disk Size: 1.8M + Message: Process 2649303 (qemu-system-x86) of user 107 dumped core. + + Stack trace of thread 2649303: + #0 0x00007ff3c32436be n/a (libc.so.6 + 0xf56be) + #1 0x000055a949c0ed05 qemu_poll_ns (qemu-system-x86_64 + 0x7b0d05) + #2 0x000055a949c0e476 main_loop_wait (qemu-system-x86_64 + 0x7b0476) + #3 0x000055a949a36d27 qemu_main_loop (qemu-system-x86_64 + 0x5d8d27) + #4 0x000055a94979e4d2 main (qemu-system-x86_64 + 0x3404d2) + #5 0x00007ff3c3175b75 n/a (libc.so.6 + 0x27b75) + #6 0x000055a9497a1f5e _start (qemu-system-x86_64 + 0x343f5e) + + Stack trace of thread 2649368: + #0 0x00007ff3c32435bf n/a (libc.so.6 + 0xf55bf) + #1 0x00007ff3c3af547c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa947c) + #2 0x00007ff3c3aa0a93 g_main_loop_run (libglib-2.0.so.0 + 0x54a93) + #3 0x00007ff3c17a727a red_worker_main.lto_priv.0 (libspice-server.so.1 + 0x5227a) + #4 0x00007ff3c3326299 start_thread (libpthread.so.0 + 0x9299) + #5 0x00007ff3c324e353 n/a (libc.so.6 + 0x100353) + + ...snip... -- 2.31.1