On Mon, Jan 17, 2022 at 04:39:09PM +0100, Peter Krempa wrote: So, a user is trying to figure out which mode is on (with only a basic knowledge of libvirt) and they need to pick a daemon of interest. I think we can improve what you wrote a little by incorporating a more generic bit with the followin: systemctl list-units -t service -t socket ... virtnwfilterd.service loaded active running Virtualization nwfilter daemon virtqemud.service loaded active running Virtualization qemu daemon ----------------------------------- ... virtinterfaced-admin.socket loaded active listening Libvirt interface admin socket virtinterfaced-ro.socket loaded active listening Libvirt interface local read-only socket virtinterfaced.socket loaded active listening Libvirt interface local socket virtlockd.socket loaded active listening Virtual machine lock manager socket virtlxcd-admin.socket loaded active listening Libvirt lxc admin socket virtlxcd-ro.socket loaded active listening Libvirt lxc local read-only socket virtlxcd.socket loaded active listening Libvirt lxc local socket virtnetworkd-admin.socket loaded active listening Libvirt network admin socket virtnetworkd-ro.socket loaded active listening Libvirt network local read-only socket virtnetworkd.socket loaded active listening Libvirt network local socket virtnodedevd.socket loaded active listening Libvirt nodedev local socket virtnwfilterd-admin.socket loaded active running Libvirt nwfilter admin socket virtnwfilterd-ro.socket loaded active running Libvirt nwfilter local read-only socket virtnwfilterd.socket loaded active running Libvirt nwfilter local socket virtproxyd.socket loaded active listening Libvirt proxy local socket virtqemud-admin.socket loaded active running Libvirt qemu admin socket virtqemud-ro.socket loaded active running Libvirt qemu local read-only socket virtqemud.socket loaded active running Libvirt qemu local socket virtsecretd.socket loaded active listening Libvirt secret local socket virtstoraged.socket loaded active listening Libvirt storage local socket If they see a bunch of virt- prefixed sockets/services, then they're running with in the modular mode. Otherwise the patch is fine. Erik

Use the heading hierarchy as generated by pandoc which we use in most of the converted XML format docs. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 3fb06bedc3..967ec10f4a 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -5,7 +5,7 @@ Debug Logs .. contents:: Turning on debug logs ---------------------- +===================== If you `report a bug <https://gitlab.com/libvirt/libvirt/-/issues/new>`__ against libvirt, in most cases you will be asked to attach debug logs. These @@ -18,10 +18,10 @@ well. How to turn on debug logs for libvirt -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- Persistent setting -^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~ The daemon configuration files location is dependent on `connection URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: @@ -71,7 +71,7 @@ However, when you are using the session mode ``qemu:///session`` or you run the ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. Runtime setting -^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~ Debugging anomalies can be very painful, especially when trying to reproduce it after the daemon restarts, since the new session can make the anomaly @@ -112,7 +112,7 @@ once you're finished debugging, just remember to save the original sets of filters and outputs and restore them at the end the same way as described above. Removing filters and outputs -'''''''''''''''''''''''''''' +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It's also possible to remove all the filters and produce an enormous log file, but it is not recommended since some of libvirt's modules can produce a large @@ -136,7 +136,7 @@ setting which depends on the host configuration, *journald* in our case: Logging outputs: 2:journald What to attach? ---------------- +=============== Now you should go and reproduce the bug. Once you're finished, attach: @@ -156,7 +156,7 @@ Now you should go and reproduce the bug. Once you're finished, attach: required. Example filter settings ------------------------ +======================= Some filter setting suggestions for debugging more specific things. Unless it's explicitly stated, these work on libvirt 4.4.0 and later. Please note that some @@ -164,7 +164,7 @@ of the filters below may not log enough information for filing a proper libvirt bug. Usually it's better to log more than less. Targeted logging for debugging QEMU VMs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------------------- Specifying only some sections allows for a targeted filter configuration which works on all versions and is sufficient for most cases. @@ -174,7 +174,7 @@ works on all versions and is sufficient for most cases. 1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util Less verbose logging for QEMU VMs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------------- Some subsystems are very noisy and usually not the culprit of the problems. They can be silenced individually for a less verbose log while still logging @@ -186,7 +186,7 @@ A permissive filter is good for development use cases. 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:* Minimalistic QEMU QMP monitor logging -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- This filter logs only QMP traffic and skips most of libvirt's messages. -- 2.34.1

Enabling the logs is the focus of this article. Decouple it from the first section. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 967ec10f4a..e0115bd23f 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -18,10 +18,10 @@ well. How to turn on debug logs for libvirt -------------------------------------- +===================================== Persistent setting -~~~~~~~~~~~~~~~~~~ +------------------ The daemon configuration files location is dependent on `connection URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: @@ -71,7 +71,7 @@ However, when you are using the session mode ``qemu:///session`` or you run the ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. Runtime setting -~~~~~~~~~~~~~~~ +--------------- Debugging anomalies can be very painful, especially when trying to reproduce it after the daemon restarts, since the new session can make the anomaly @@ -112,7 +112,7 @@ once you're finished debugging, just remember to save the original sets of filters and outputs and restore them at the end the same way as described above. Removing filters and outputs -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's also possible to remove all the filters and produce an enormous log file, but it is not recommended since some of libvirt's modules can produce a large -- 2.34.1

Move out the settings required to pre libvirt-4.4.0 deployments into a separate section so that the main point is not cluttered by now mostly irrelevant settings. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 48 +++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index e0115bd23f..cba57e020d 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -33,14 +33,6 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: :: - # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 SEE BELOW! # - log_level = 1 - log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" - -:: - - # PREFERRED SETTINGS AFTER LIBVIRT 4.4.0 # log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" log_outputs="1:file:/var/log/libvirt/libvirtd.log" @@ -51,12 +43,10 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: systemctl restart libvirtd.service -In the config variables above, we have set logging level to 1 (debug level), set -some filters (to filter out noise), e.g. from rpc only warnings (=level 3) and -above will be reported. The logs are saved into -``/var/log/libvirt/libvirtd.log``. Since libvirt **4.4.0** log filters support -shell globbing, therefore the usage of ``log_level`` is considered deprecated in -favour of pure usage of ``log_filters``. + +*Note:* Libvirt prior to the ``libvirt-4.4.0`` release didn't support globbing +patterns and thus requires more configuration. See +`Legacy (pre-4.4.0) libvirt daemon logging configuration`_. In case you want to get the client logs, you need to set this environment variable: @@ -91,12 +81,6 @@ own set of filters: :: - ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## - # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1:nodedev 1:qemu" - -:: - - ## CURRENT APPROACH USING SHELL GLOBBING ## # virt-admin daemon-log-filters "3:remote 4:util.json 4:rpc 1:*" Analogically, the same procedure can be performed with log outputs: @@ -193,3 +177,27 @@ This filter logs only QMP traffic and skips most of libvirt's messages. :: 2:qemu.qemu_monitor 3:* + + +Legacy (pre-4.4.0) libvirt daemon logging configuration +======================================================= + +Old libvirt versions didn't support globbing (e.g. ``1:*``) to configure logging +wildcarts, thus it's required to explicitly set logging level to 1 (debug level) +via the ``log_level`` setting and then filter out the noise via targetted log +filters. + +:: + + # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 + log_level = 1 + log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc" + log_outputs="1:file:/var/log/libvirt/libvirtd.log" + + +Or using ``virt-admin``: + +:: + + ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## + # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1:nodedev 1:qemu" -- 2.34.1

Outline what the given settings influence. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index cba57e020d..9ba39b9f85 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -17,6 +17,73 @@ Moreover, libvirt catches stderr of all running domains. These can be useful as well. +Logging settings in libvirt +=========================== + +Log levels +---------- + +Libvirt produces log entries in 4 priority levels; higher priority level meaning +more severe log entries, which in turn means less of them. The log level are +used by `Log outputs`_ and `Log filters`_ below): + + * ``1: DEBUG`` + * ``2: INFO`` + * ``3: WARNING`` + * ``4: ERROR`` + +For debugging it's necessary to capture ``DEBUG`` level entries as the name +implies. + +Log outputs +----------- + +Log outputs describe where the log messages are being recorded. The outputs +are described by a space-separated list of tuples in the following format: + +:: + + level:output + +``level`` refers to the minimum priority level of entries recorded in the output. + +``output`` is one of the following: + + ``file:FILENAME`` + Logging messages are appended to FILENAME. + + ``journald`` + Logging goes to the ``journald`` logging daemon. + + ``stderr`` + Logging goes to the standard error output stream of the libvirt daemon. + + ``syslog:name`` + Logging goes to syslogd. ``name`` is used to identify the entries. + +The default output on systems running ``journald`` is ``3:journald``. Note that +``journald`` can trottle the amount of logs per process so for capturing debug +logs of libvirt daemons a file output should be used in addition to the output +to the logging daemon e.g.: + +:: + + "1:file:/var/log/libvirt/libvirtd.log 3:journald" + + +Log filters +----------- + +Log filters allow to avoid logging of messages which are not relevant to the +cause. The log filters is a space-separated list of tuples in the format +``level:identifer`` and limits messages which match ``identifier`` to the +configured log level. Identifier supports globbing patterns such as ``*``. + +As ``identifier`` is based on internal naming of modules, the best approach is +to select one of `Example filter settings`_. It's also always better to have +more information than lack something important. + + How to turn on debug logs for libvirt ===================================== -- 2.34.1

Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 9ba39b9f85..f353cf7de5 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -84,7 +84,7 @@ to select one of `Example filter settings`_. It's also always better to have more information than lack something important. -How to turn on debug logs for libvirt +Libvirt daemons logging configuration ===================================== Persistent setting @@ -115,14 +115,6 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: patterns and thus requires more configuration. See `Legacy (pre-4.4.0) libvirt daemon logging configuration`_. -In case you want to get the client logs, you need to set this environment -variable: - -:: - - export LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" - - However, when you are using the session mode ``qemu:///session`` or you run the ``libvirtd`` as unprivileged user you will find configuration file under ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. @@ -186,6 +178,23 @@ setting which depends on the host configuration, *journald* in our case: # virt-admin daemon-log-outputs "" Logging outputs: 2:journald + +Client library logging +====================== + +By default the client library doesn't produce any logs and usually it's not +needed as in most scenarios the client library doesn't do anything interesting. + +In case you want to get the client logs, logging is controlled via +environment variables ``LIBVIRT_LOG_OUTPUTS`` and ``LIBVIRT_LOG_FILTERS``. +Generally when client logs are needed don't filter them: + +:: + + export LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" + + + What to attach? =============== -- 2.34.1

It's a tool name so use backticks to format it in monospace. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index f353cf7de5..fc75ef14a4 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -125,10 +125,11 @@ Runtime setting Debugging anomalies can be very painful, especially when trying to reproduce it after the daemon restarts, since the new session can make the anomaly "disappear". Therefore, it's possible to enable the debug logs during runtime -using libvirt administration API. To use it conveniently, there's a virt-admin -client provided by the libvirt-admin package. Use the package manager provided -by your distribution to install this package. Once you have it installed, run -the following as root to see the set of log filters currently being active: +using libvirt administration API. To use it conveniently, there's the +``virt-admin`` client provided by the ``libvirt-admin`` package. Use the +package manager provided by your distribution to install this package. Once you +have it installed, run the following as root to see the set of log filters +currently being active: :: -- 2.34.1

Modular daemons are now the default in many new installations, thus we need to include steps how to determine that modular daemons are used and modular-daemon specific locations for the config files and admin URIs. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/debuglogs.rst | 94 +++++++++++++++++++++++++++++++--------- 1 file changed, 73 insertions(+), 21 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index fc75ef14a4..ea7c2c1d52 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -87,38 +87,87 @@ more information than lack something important. Libvirt daemons logging configuration ===================================== -Persistent setting ------------------- +Libvirt daemons can be configured either via a config file or via the +administration API. The configuration location depends on multiple factors. -The daemon configuration files location is dependent on `connection -URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: +Session vs. system daemons +-------------------------- +The libvirt daemons run either on ``system`` level or on ``session`` (user) +level, depending on the configuration of the host and the available permission +levels. +The `connection URI <https://libvirt.org/uri.html>`__ influences which daemon +the client is communicating with. -- open ``/etc/libvirt/libvirtd.conf`` in your favourite editor -- find & replace, or set these variables: +``system`` daemon e.g ``qemu:///system`` URI -:: + * config files are usually placed in ``/etc/libvirt/`` - log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" + * ``virt-admin`` URI ending in ``/system`` should be used. -- save and exit -- restart libvirtd service +``session`` daemon e.g. ``qemu:///session`` URI -:: + * config files are usually placed in ``$XDG_CONFIG_HOME/libvirt/`` directory + + * ``virt-admin`` URI ending in ``/session`` should be used + +Modular vs. monolithic daemons +------------------------------ + +While the config file name for the monolithic daemon is simple, in case when +modular daemons are used, there's a lot of options. + +All libvirt daemons support the logging configuration per this document, but +in most cases it's required to configure logging for the appropriate. + +Refer to `documentation about daemons <../daemons.html#checking-whether-modular-monolithic-mode-is-in-use>`__ +to figure out which is in use by your system. + + Modular daemons + + The configuration of modular daemons is in file named after the daemon. E.g. + for ``qemu:///system`` connection this is the ``virtqemud`` daemon and + correspondingly: + + * ``virtqemud.conf`` config file is used + + * ``virtqemud:///system`` or ``virtqemud:///session`` admin URI is used - systemctl restart libvirtd.service + Monolithic daemon + + * ``libvirtd.conf`` config file is used + + * ``libvirtd:///system`` or ``libvirtd:///session`` admin URI is used + when the modular qemu hypervisor driver ``virtqemud`` + +Persistent setting +------------------ + +- open the appropriate daemon config file in your favourite editor :: + + /etc/libvirt/virtqemud.conf + /etc/libvirt/libvirtd.conf + $XDG_CONFIG_HOME/libvirt/libvirtd.conf + $XDG_CONFIG_HOME/libvirt/virtqemud.conf + +- find & replace, or set the apropriate `Log outputs`_ and `Log filters`_, e.g :: + + log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" + log_outputs="1:file:/var/log/libvirt/libvirtd.log" + +- save and exit +- restart the corresponding service/daemon e.g. :: + + systemctl restart virtqemud.socket + systemctl restart libvirtd.socket + systemctl restart libvirtd.service *Note:* Libvirt prior to the ``libvirt-4.4.0`` release didn't support globbing patterns and thus requires more configuration. See `Legacy (pre-4.4.0) libvirt daemon logging configuration`_. -However, when you are using the session mode ``qemu:///session`` or you run the -``libvirtd`` as unprivileged user you will find configuration file under -``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. - Runtime setting --------------- @@ -127,13 +176,16 @@ after the daemon restarts, since the new session can make the anomaly "disappear". Therefore, it's possible to enable the debug logs during runtime using libvirt administration API. To use it conveniently, there's the ``virt-admin`` client provided by the ``libvirt-admin`` package. Use the -package manager provided by your distribution to install this package. Once you -have it installed, run the following as root to see the set of log filters -currently being active: +package manager provided by your distribution to install this package. + +**Important**: Substitute ``virt-admin -c $ADMIN_URI`` per above guidance in +place of ``virt-admin`` in the examples below if needed. + +The following command allows to query the list of log filters currently active: :: - # virt-admin daemon-log-filters + # virt-admin daemon-log-filters Logging filters: 3:remote 4:util.json 4:rpc In order to change this set, run the same command as root, this time with your -- 2.34.1