[PATCH 0/8] kbase: debuglogs: Include guidance how to setup logging with modular daemons
Modular daemons are the default in many cases yet, one of our most frequented documents neglects to describe the intricacies of the setup. Browsable result is available at: https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1983126893/artifacts/website/daem... https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1983126893/artifacts/website/kbas... Sorry for TLS warnings. Gitlab didn't really think through allowing users to pick username with a dot and I refuse to change it. Peter Krempa (8): docs: daemons: Add section on figuring out whether modular or monolithic daemon is in use kbase: debuglogs: Use common heading hierarchy kbase: debuglogs: Un-nest section on enabling debug logs kbase: debuglogs: Section off the settings for legacy libvirt kbase: debuglogs: Add a section describing log outputs and filters kbase: debuglogs: Extract client library logging setup kbase: debuglogs: Emphasise 'virt-admin' kbase: debuglogs: Fix for modular daemons docs/daemons.rst | 47 ++++++++ docs/kbase/debuglogs.rst | 239 ++++++++++++++++++++++++++++++--------- 2 files changed, 235 insertions(+), 51 deletions(-) -- 2.34.1
Since we are at a transition period where some users may be running monolithic libvirtd and others already the modular topology we need a section that allows users to figure out which is in use. This will be particularly important in the document about enabling logging, as the active log file depends on which daemon is in use. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/daemons.rst | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/daemons.rst b/docs/daemons.rst index c8ae3b0cef..1446c1f92c 100644 --- a/docs/daemons.rst +++ b/docs/daemons.rst @@ -435,6 +435,53 @@ host first. $ systemctl enable virtproxyd-tls.socket $ systemctl start virtproxyd-tls.socket +Checking whether modular/monolithic mode is in use +================================================== + +To determine whether modular or monolithic mode is in use on a host running +``systemd`` as the init system you can take the following steps: + +#. Check whether the modular daemon infrastructure is in use + + First check whether the modular daemon you are interested in is running: + + #. Check ``.socket`` for socket activated services + + :: + + # systemctl is-active virtqemud.socket + active + + #. Check ``.service`` for always-running daemons + + :: + + # systemctl is-active virtqemud.service + active + + If either of the above is ``active`` your system is using the modular daemons. + +#. Check whether the monolithic daemon is in use + + #. Check ``libvirtd.socket`` + + :: + + # systemctl is-active libvirtd.socket + active + + #. Check ``libvirtd.service`` for always-running daemon + + :: + + # systemctl is-active libvirtd.service + active + + If either of the above is ``active`` your system is using the monolithic + daemon. + +#. To determine which of the above will be in use on the next boot of the system, + substitute ``is-enabled`` for ``is-active`` in the above examples. Proxy daemon ============ -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:09PM +0100, Peter Krempa wrote:
Since we are at a transition period where some users may be running monolithic libvirtd and others already the modular topology we need a section that allows users to figure out which is in use.
This will be particularly important in the document about enabling logging, as the active log file depends on which daemon is in use.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/daemons.rst | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
diff --git a/docs/daemons.rst b/docs/daemons.rst index c8ae3b0cef..1446c1f92c 100644 --- a/docs/daemons.rst +++ b/docs/daemons.rst @@ -435,6 +435,53 @@ host first. $ systemctl enable virtproxyd-tls.socket $ systemctl start virtproxyd-tls.socket
+Checking whether modular/monolithic mode is in use +================================================== + +To determine whether modular or monolithic mode is in use on a host running +``systemd`` as the init system you can take the following steps: + +#. Check whether the modular daemon infrastructure is in use + + First check whether the modular daemon you are interested in is running:
So, a user is trying to figure out which mode is on (with only a basic knowledge of libvirt) and they need to pick a daemon of interest. I think we can improve what you wrote a little by incorporating a more generic bit with the followin: systemctl list-units -t service -t socket ... virtnwfilterd.service loaded active running Virtualization nwfilter daemon virtqemud.service loaded active running Virtualization qemu daemon ----------------------------------- ... virtinterfaced-admin.socket loaded active listening Libvirt interface admin socket virtinterfaced-ro.socket loaded active listening Libvirt interface local read-only socket virtinterfaced.socket loaded active listening Libvirt interface local socket virtlockd.socket loaded active listening Virtual machine lock manager socket virtlxcd-admin.socket loaded active listening Libvirt lxc admin socket virtlxcd-ro.socket loaded active listening Libvirt lxc local read-only socket virtlxcd.socket loaded active listening Libvirt lxc local socket virtnetworkd-admin.socket loaded active listening Libvirt network admin socket virtnetworkd-ro.socket loaded active listening Libvirt network local read-only socket virtnetworkd.socket loaded active listening Libvirt network local socket virtnodedevd.socket loaded active listening Libvirt nodedev local socket virtnwfilterd-admin.socket loaded active running Libvirt nwfilter admin socket virtnwfilterd-ro.socket loaded active running Libvirt nwfilter local read-only socket virtnwfilterd.socket loaded active running Libvirt nwfilter local socket virtproxyd.socket loaded active listening Libvirt proxy local socket virtqemud-admin.socket loaded active running Libvirt qemu admin socket virtqemud-ro.socket loaded active running Libvirt qemu local read-only socket virtqemud.socket loaded active running Libvirt qemu local socket virtsecretd.socket loaded active listening Libvirt secret local socket virtstoraged.socket loaded active listening Libvirt storage local socket If they see a bunch of virt- prefixed sockets/services, then they're running with in the modular mode. Otherwise the patch is fine. Erik
+ + #. Check ``.socket`` for socket activated services + + :: + + # systemctl is-active virtqemud.socket + active + + #. Check ``.service`` for always-running daemons + + :: + + # systemctl is-active virtqemud.service + active + + If either of the above is ``active`` your system is using the modular daemons. + +#. Check whether the monolithic daemon is in use + + #. Check ``libvirtd.socket`` + + :: + + # systemctl is-active libvirtd.socket + active + + #. Check ``libvirtd.service`` for always-running daemon + + :: + + # systemctl is-active libvirtd.service + active + + If either of the above is ``active`` your system is using the monolithic + daemon. + +#. To determine which of the above will be in use on the next boot of the system, + substitute ``is-enabled`` for ``is-active`` in the above examples.
Proxy daemon ============ -- 2.34.1
On Tue, Jan 18, 2022 at 16:22:08 +0100, Erik Skultety wrote:
On Mon, Jan 17, 2022 at 04:39:09PM +0100, Peter Krempa wrote:
Since we are at a transition period where some users may be running monolithic libvirtd and others already the modular topology we need a section that allows users to figure out which is in use.
This will be particularly important in the document about enabling logging, as the active log file depends on which daemon is in use.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/daemons.rst | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
diff --git a/docs/daemons.rst b/docs/daemons.rst index c8ae3b0cef..1446c1f92c 100644 --- a/docs/daemons.rst +++ b/docs/daemons.rst @@ -435,6 +435,53 @@ host first. $ systemctl enable virtproxyd-tls.socket $ systemctl start virtproxyd-tls.socket
+Checking whether modular/monolithic mode is in use +================================================== + +To determine whether modular or monolithic mode is in use on a host running +``systemd`` as the init system you can take the following steps: + +#. Check whether the modular daemon infrastructure is in use + + First check whether the modular daemon you are interested in is running:
So, a user is trying to figure out which mode is on (with only a basic knowledge of libvirt) and they need to pick a daemon of interest. I think we can improve what you wrote a little by incorporating a more generic bit with the followin:
The 'Modular driver daemons' section of this document actually enumerates which daemons libvirt provides, so I can link to that ...
systemctl list-units -t service -t socket
... virtnwfilterd.service loaded active running Virtualization nwfilter daemon virtqemud.service loaded active running Virtualization qemu daemon ----------------------------------- ... virtinterfaced-admin.socket loaded active listening Libvirt interface admin socket virtinterfaced-ro.socket loaded active listening Libvirt interface local read-only socket virtinterfaced.socket loaded active listening Libvirt interface local socket virtlockd.socket loaded active listening Virtual machine lock manager socket virtlxcd-admin.socket loaded active listening Libvirt lxc admin socket virtlxcd-ro.socket loaded active listening Libvirt lxc local read-only socket virtlxcd.socket loaded active listening Libvirt lxc local socket virtnetworkd-admin.socket loaded active listening Libvirt network admin socket virtnetworkd-ro.socket loaded active listening Libvirt network local read-only socket virtnetworkd.socket loaded active listening Libvirt network local socket virtnodedevd.socket loaded active listening Libvirt nodedev local socket virtnwfilterd-admin.socket loaded active running Libvirt nwfilter admin socket virtnwfilterd-ro.socket loaded active running Libvirt nwfilter local read-only socket virtnwfilterd.socket loaded active running Libvirt nwfilter local socket virtproxyd.socket loaded active listening Libvirt proxy local socket virtqemud-admin.socket loaded active running Libvirt qemu admin socket virtqemud-ro.socket loaded active running Libvirt qemu local read-only socket virtqemud.socket loaded active running Libvirt qemu local socket virtsecretd.socket loaded active listening Libvirt secret local socket virtstoraged.socket loaded active listening Libvirt storage local socket
If they see a bunch of virt- prefixed sockets/services, then they're running with in the modular mode.
This seems a bit excessive though.
Otherwise the patch is fine.
Use the heading hierarchy as generated by pandoc which we use in most of the converted XML format docs. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 3fb06bedc3..967ec10f4a 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -5,7 +5,7 @@ Debug Logs .. contents:: Turning on debug logs ---------------------- +===================== If you `report a bug <https://gitlab.com/libvirt/libvirt/-/issues/new>`__ against libvirt, in most cases you will be asked to attach debug logs. These @@ -18,10 +18,10 @@ well. How to turn on debug logs for libvirt -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- Persistent setting -^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~ The daemon configuration files location is dependent on `connection URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: @@ -71,7 +71,7 @@ However, when you are using the session mode ``qemu:///session`` or you run the ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. Runtime setting -^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~ Debugging anomalies can be very painful, especially when trying to reproduce it after the daemon restarts, since the new session can make the anomaly @@ -112,7 +112,7 @@ once you're finished debugging, just remember to save the original sets of filters and outputs and restore them at the end the same way as described above. Removing filters and outputs -'''''''''''''''''''''''''''' +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It's also possible to remove all the filters and produce an enormous log file, but it is not recommended since some of libvirt's modules can produce a large @@ -136,7 +136,7 @@ setting which depends on the host configuration, *journald* in our case: Logging outputs: 2:journald What to attach? ---------------- +=============== Now you should go and reproduce the bug. Once you're finished, attach: @@ -156,7 +156,7 @@ Now you should go and reproduce the bug. Once you're finished, attach: required. Example filter settings ------------------------ +======================= Some filter setting suggestions for debugging more specific things. Unless it's explicitly stated, these work on libvirt 4.4.0 and later. Please note that some @@ -164,7 +164,7 @@ of the filters below may not log enough information for filing a proper libvirt bug. Usually it's better to log more than less. Targeted logging for debugging QEMU VMs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------------------- Specifying only some sections allows for a targeted filter configuration which works on all versions and is sufficient for most cases. @@ -174,7 +174,7 @@ works on all versions and is sufficient for most cases. 1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util Less verbose logging for QEMU VMs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------------- Some subsystems are very noisy and usually not the culprit of the problems. They can be silenced individually for a less verbose log while still logging @@ -186,7 +186,7 @@ A permissive filter is good for development use cases. 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:* Minimalistic QEMU QMP monitor logging -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- This filter logs only QMP traffic and skips most of libvirt's messages. -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:10PM +0100, Peter Krempa wrote:
Use the heading hierarchy as generated by pandoc which we use in most of the converted XML format docs.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>
Enabling the logs is the focus of this article. Decouple it from the first section. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 967ec10f4a..e0115bd23f 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -18,10 +18,10 @@ well. How to turn on debug logs for libvirt -------------------------------------- +===================================== Persistent setting -~~~~~~~~~~~~~~~~~~ +------------------ The daemon configuration files location is dependent on `connection URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: @@ -71,7 +71,7 @@ However, when you are using the session mode ``qemu:///session`` or you run the ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. Runtime setting -~~~~~~~~~~~~~~~ +--------------- Debugging anomalies can be very painful, especially when trying to reproduce it after the daemon restarts, since the new session can make the anomaly @@ -112,7 +112,7 @@ once you're finished debugging, just remember to save the original sets of filters and outputs and restore them at the end the same way as described above. Removing filters and outputs -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's also possible to remove all the filters and produce an enormous log file, but it is not recommended since some of libvirt's modules can produce a large -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:11PM +0100, Peter Krempa wrote:
Enabling the logs is the focus of this article. Decouple it from the first section.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>
Move out the settings required to pre libvirt-4.4.0 deployments into a separate section so that the main point is not cluttered by now mostly irrelevant settings. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 48 +++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index e0115bd23f..cba57e020d 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -33,14 +33,6 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: :: - # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 SEE BELOW! # - log_level = 1 - log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" - -:: - - # PREFERRED SETTINGS AFTER LIBVIRT 4.4.0 # log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" log_outputs="1:file:/var/log/libvirt/libvirtd.log" @@ -51,12 +43,10 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: systemctl restart libvirtd.service -In the config variables above, we have set logging level to 1 (debug level), set -some filters (to filter out noise), e.g. from rpc only warnings (=level 3) and -above will be reported. The logs are saved into -``/var/log/libvirt/libvirtd.log``. Since libvirt **4.4.0** log filters support -shell globbing, therefore the usage of ``log_level`` is considered deprecated in -favour of pure usage of ``log_filters``. + +*Note:* Libvirt prior to the ``libvirt-4.4.0`` release didn't support globbing +patterns and thus requires more configuration. See +`Legacy (pre-4.4.0) libvirt daemon logging configuration`_. In case you want to get the client logs, you need to set this environment variable: @@ -91,12 +81,6 @@ own set of filters: :: - ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## - # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1:nodedev 1:qemu" - -:: - - ## CURRENT APPROACH USING SHELL GLOBBING ## # virt-admin daemon-log-filters "3:remote 4:util.json 4:rpc 1:*" Analogically, the same procedure can be performed with log outputs: @@ -193,3 +177,27 @@ This filter logs only QMP traffic and skips most of libvirt's messages. :: 2:qemu.qemu_monitor 3:* + + +Legacy (pre-4.4.0) libvirt daemon logging configuration +======================================================= + +Old libvirt versions didn't support globbing (e.g. ``1:*``) to configure logging +wildcarts, thus it's required to explicitly set logging level to 1 (debug level) +via the ``log_level`` setting and then filter out the noise via targetted log +filters. + +:: + + # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 + log_level = 1 + log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc" + log_outputs="1:file:/var/log/libvirt/libvirtd.log" + + +Or using ``virt-admin``: + +:: + + ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## + # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1:nodedev 1:qemu" -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:12PM +0100, Peter Krempa wrote:
Move out the settings required to pre libvirt-4.4.0 deployments into a separate section so that the main point is not cluttered by now mostly irrelevant settings.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 48 +++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 20 deletions(-)
diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index e0115bd23f..cba57e020d 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -33,14 +33,6 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``:
::
- # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 SEE BELOW! # - log_level = 1 - log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" - -:: - - # PREFERRED SETTINGS AFTER LIBVIRT 4.4.0 # log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" log_outputs="1:file:/var/log/libvirt/libvirtd.log"
@@ -51,12 +43,10 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``:
systemctl restart libvirtd.service
-In the config variables above, we have set logging level to 1 (debug level), set -some filters (to filter out noise), e.g. from rpc only warnings (=level 3) and -above will be reported. The logs are saved into -``/var/log/libvirt/libvirtd.log``. Since libvirt **4.4.0** log filters support -shell globbing, therefore the usage of ``log_level`` is considered deprecated in -favour of pure usage of ``log_filters``. + +*Note:* Libvirt prior to the ``libvirt-4.4.0`` release didn't support globbing +patterns and thus requires more configuration. See +`Legacy (pre-4.4.0) libvirt daemon logging configuration`_.
In case you want to get the client logs, you need to set this environment variable: @@ -91,12 +81,6 @@ own set of filters:
::
- ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## - # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1:nodedev 1:qemu" - -:: - - ## CURRENT APPROACH USING SHELL GLOBBING ## # virt-admin daemon-log-filters "3:remote 4:util.json 4:rpc 1:*"
Analogically, the same procedure can be performed with log outputs: @@ -193,3 +177,27 @@ This filter logs only QMP traffic and skips most of libvirt's messages. ::
2:qemu.qemu_monitor 3:* + + +Legacy (pre-4.4.0) libvirt daemon logging configuration +=======================================================
This section should be re-parented under the "Libvirt daemons logging configuration" as the last section - the reasoning is that it is still related to the section about the daemon configuration.
+ +Old libvirt versions didn't support globbing (e.g. ``1:*``) to configure logging +wildcarts, thus it's required to explicitly set logging level to 1 (debug level)
s/wildcarts// # (globbing is all about wildcards...)
+via the ``log_level`` setting and then filter out the noise via targetted log +filters.
s/via/with/g s/targetted/a tailored log_filters string With the adjustments: Reviewed-by: Erik Skultety <eskultet@redhat.com>
+ +:: + + # LEGACY SETTINGS PRIOR LIBVIRT 4.4.0 + log_level = 1 + log_filters="1:qemu 3:remote 4:event 3:util.json 3:rpc" + log_outputs="1:file:/var/log/libvirt/libvirtd.log" + + +Or using ``virt-admin``: + +:: + + ## LEGACY APPROACH ENUMERATING ALL THE DESIRED MODULES ## + # virt-admin daemon-log-filters "1:util 1:libvirt 1:storage 1:network 1:nodedev 1:qemu" -- 2.34.1
Outline what the given settings influence. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index cba57e020d..9ba39b9f85 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -17,6 +17,73 @@ Moreover, libvirt catches stderr of all running domains. These can be useful as well. +Logging settings in libvirt +=========================== + +Log levels +---------- + +Libvirt produces log entries in 4 priority levels; higher priority level meaning +more severe log entries, which in turn means less of them. The log level are +used by `Log outputs`_ and `Log filters`_ below): + + * ``1: DEBUG`` + * ``2: INFO`` + * ``3: WARNING`` + * ``4: ERROR`` + +For debugging it's necessary to capture ``DEBUG`` level entries as the name +implies. + +Log outputs +----------- + +Log outputs describe where the log messages are being recorded. The outputs +are described by a space-separated list of tuples in the following format: + +:: + + level:output + +``level`` refers to the minimum priority level of entries recorded in the output. + +``output`` is one of the following: + + ``file:FILENAME`` + Logging messages are appended to FILENAME. + + ``journald`` + Logging goes to the ``journald`` logging daemon. + + ``stderr`` + Logging goes to the standard error output stream of the libvirt daemon. + + ``syslog:name`` + Logging goes to syslogd. ``name`` is used to identify the entries. + +The default output on systems running ``journald`` is ``3:journald``. Note that +``journald`` can trottle the amount of logs per process so for capturing debug +logs of libvirt daemons a file output should be used in addition to the output +to the logging daemon e.g.: + +:: + + "1:file:/var/log/libvirt/libvirtd.log 3:journald" + + +Log filters +----------- + +Log filters allow to avoid logging of messages which are not relevant to the +cause. The log filters is a space-separated list of tuples in the format +``level:identifer`` and limits messages which match ``identifier`` to the +configured log level. Identifier supports globbing patterns such as ``*``. + +As ``identifier`` is based on internal naming of modules, the best approach is +to select one of `Example filter settings`_. It's also always better to have +more information than lack something important. + + How to turn on debug logs for libvirt ===================================== -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:13PM +0100, Peter Krempa wrote:
Outline what the given settings influence.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index cba57e020d..9ba39b9f85 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -17,6 +17,73 @@ Moreover, libvirt catches stderr of all running domains. These can be useful as well.
+Logging settings in libvirt +=========================== + +Log levels +---------- + +Libvirt produces log entries in 4 priority levels; higher priority level meaning
"Libvirt log messages are classified into 4 priority levels;... also "the higher the priority level, the less is the volume of produced messages"
+more severe log entries, which in turn means less of them. The log level are +used by `Log outputs`_ and `Log filters`_ below):
"The log level setting is controlled by the 'log_filters' and 'log_outputs' settings explained in the `Log outputs`_ and `Log filters`_ sections respectively."
+ + * ``1: DEBUG`` + * ``2: INFO`` + * ``3: WARNING`` + * ``4: ERROR`` + +For debugging it's necessary to capture ``DEBUG`` level entries as the name
s/capture/capture the
+implies. + +Log outputs +----------- + +Log outputs describe where the log messages are being recorded. The outputs +are described by a space-separated list of tuples in the following format: + +:: + + level:output + +``level`` refers to the minimum priority level of entries recorded in the output. + +``output`` is one of the following: + + ``file:FILENAME`` + Logging messages are appended to FILENAME. + + ``journald`` + Logging goes to the ``journald`` logging daemon. + + ``stderr`` + Logging goes to the standard error output stream of the libvirt daemon. + + ``syslog:name`` + Logging goes to syslogd. ``name`` is used to identify the entries. + +The default output on systems running ``journald`` is ``3:journald``. Note that +``journald`` can trottle the amount of logs per process so for capturing debug
s/trottle/throttle s/for capturing/in order to capture
+logs of libvirt daemons a file output should be used in addition to the output +to the logging daemon e.g.:
"logs of a libvirt daemon should go to a file instead (in addition to the original logging daemon), e.g.:"
+ +:: + + "1:file:/var/log/libvirt/libvirtd.log 3:journald" + + +Log filters +----------- + +Log filters allow to avoid logging of messages which are not relevant to the
"Log filters, as the name suggest, help filtering out messages which are irrelevant to the cause."
+cause. The log filters is a space-separated list of tuples in the format +``level:identifer`` and limits messages which match ``identifier`` to the +configured log level. Identifier supports globbing patterns such as ``*``.
"...list of tuples using the ``level:identifier`` format. Each filter defined this way will then limit messages coming from a module matching the ``identifier`` pattern (accepts globs too) to the given ``level``."
+ +As ``identifier`` is based on internal naming of modules, the best approach is
s/best approach/preferred way of configuring your filters is to start with the `Example filter settings`_.
+to select one of `Example filter settings`_. It's also always better to have +more information than lack something important.
"The rule of thumb here is to have more logs rather than less and miss something important". Only stylistic nitpicks....: Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 9ba39b9f85..f353cf7de5 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -84,7 +84,7 @@ to select one of `Example filter settings`_. It's also always better to have more information than lack something important. -How to turn on debug logs for libvirt +Libvirt daemons logging configuration ===================================== Persistent setting @@ -115,14 +115,6 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: patterns and thus requires more configuration. See `Legacy (pre-4.4.0) libvirt daemon logging configuration`_. -In case you want to get the client logs, you need to set this environment -variable: - -:: - - export LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" - - However, when you are using the session mode ``qemu:///session`` or you run the ``libvirtd`` as unprivileged user you will find configuration file under ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. @@ -186,6 +178,23 @@ setting which depends on the host configuration, *journald* in our case: # virt-admin daemon-log-outputs "" Logging outputs: 2:journald + +Client library logging +====================== + +By default the client library doesn't produce any logs and usually it's not +needed as in most scenarios the client library doesn't do anything interesting. + +In case you want to get the client logs, logging is controlled via +environment variables ``LIBVIRT_LOG_OUTPUTS`` and ``LIBVIRT_LOG_FILTERS``. +Generally when client logs are needed don't filter them: + +:: + + export LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" + + + What to attach? =============== -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:14PM +0100, Peter Krempa wrote:
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index 9ba39b9f85..f353cf7de5 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -84,7 +84,7 @@ to select one of `Example filter settings`_. It's also always better to have more information than lack something important.
-How to turn on debug logs for libvirt +Libvirt daemons logging configuration =====================================
Persistent setting @@ -115,14 +115,6 @@ URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: patterns and thus requires more configuration. See `Legacy (pre-4.4.0) libvirt daemon logging configuration`_.
-In case you want to get the client logs, you need to set this environment -variable: - -:: - - export LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" - - However, when you are using the session mode ``qemu:///session`` or you run the ``libvirtd`` as unprivileged user you will find configuration file under ``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. @@ -186,6 +178,23 @@ setting which depends on the host configuration, *journald* in our case: # virt-admin daemon-log-outputs "" Logging outputs: 2:journald
+ +Client library logging +====================== + +By default the client library doesn't produce any logs and usually it's not +needed as in most scenarios the client library doesn't do anything interesting.
maybe better worded as: "usually it's not very interesting on its own anyway."
+ +In case you want to get the client logs, logging is controlled via +environment variables ``LIBVIRT_LOG_OUTPUTS`` and ``LIBVIRT_LOG_FILTERS``.
'via the ``LIBVIRT_LOG_OUTPUTS`` and ``LIBVIRT_LOG_FILTERS`` environment variables.'
+Generally when client logs are needed don't filter them:
s/needed don't/needed make sure you don't/ Reviewed-by: Erik Skultety <eskultet@redhat.com>
+ +:: + + export LIBVIRT_LOG_OUTPUTS="1:file:/tmp/libvirt_client.log" + + + What to attach? ===============
-- 2.34.1
It's a tool name so use backticks to format it in monospace. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index f353cf7de5..fc75ef14a4 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -125,10 +125,11 @@ Runtime setting Debugging anomalies can be very painful, especially when trying to reproduce it after the daemon restarts, since the new session can make the anomaly "disappear". Therefore, it's possible to enable the debug logs during runtime -using libvirt administration API. To use it conveniently, there's a virt-admin -client provided by the libvirt-admin package. Use the package manager provided -by your distribution to install this package. Once you have it installed, run -the following as root to see the set of log filters currently being active: +using libvirt administration API. To use it conveniently, there's the +``virt-admin`` client provided by the ``libvirt-admin`` package. Use the +package manager provided by your distribution to install this package. Once you +have it installed, run the following as root to see the set of log filters +currently being active: :: -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:15PM +0100, Peter Krempa wrote:
It's a tool name so use backticks to format it in monospace.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- Reviewed-by: Erik Skultety <eskultet@redhat.com>
Modular daemons are now the default in many new installations, thus we need to include steps how to determine that modular daemons are used and modular-daemon specific locations for the config files and admin URIs. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 94 +++++++++++++++++++++++++++++++--------- 1 file changed, 73 insertions(+), 21 deletions(-) diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index fc75ef14a4..ea7c2c1d52 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -87,38 +87,87 @@ more information than lack something important. Libvirt daemons logging configuration ===================================== -Persistent setting ------------------- +Libvirt daemons can be configured either via a config file or via the +administration API. The configuration location depends on multiple factors. -The daemon configuration files location is dependent on `connection -URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: +Session vs. system daemons +-------------------------- +The libvirt daemons run either on ``system`` level or on ``session`` (user) +level, depending on the configuration of the host and the available permission +levels. +The `connection URI <https://libvirt.org/uri.html>`__ influences which daemon +the client is communicating with. -- open ``/etc/libvirt/libvirtd.conf`` in your favourite editor -- find & replace, or set these variables: +``system`` daemon e.g ``qemu:///system`` URI -:: + * config files are usually placed in ``/etc/libvirt/`` - log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" + * ``virt-admin`` URI ending in ``/system`` should be used. -- save and exit -- restart libvirtd service +``session`` daemon e.g. ``qemu:///session`` URI -:: + * config files are usually placed in ``$XDG_CONFIG_HOME/libvirt/`` directory + + * ``virt-admin`` URI ending in ``/session`` should be used + +Modular vs. monolithic daemons +------------------------------ + +While the config file name for the monolithic daemon is simple, in case when +modular daemons are used, there's a lot of options. + +All libvirt daemons support the logging configuration per this document, but +in most cases it's required to configure logging for the appropriate. + +Refer to `documentation about daemons <../daemons.html#checking-whether-modular-monolithic-mode-is-in-use>`__ +to figure out which is in use by your system. + + Modular daemons + + The configuration of modular daemons is in file named after the daemon. E.g. + for ``qemu:///system`` connection this is the ``virtqemud`` daemon and + correspondingly: + + * ``virtqemud.conf`` config file is used + + * ``virtqemud:///system`` or ``virtqemud:///session`` admin URI is used - systemctl restart libvirtd.service + Monolithic daemon + + * ``libvirtd.conf`` config file is used + + * ``libvirtd:///system`` or ``libvirtd:///session`` admin URI is used + when the modular qemu hypervisor driver ``virtqemud`` + +Persistent setting +------------------ + +- open the appropriate daemon config file in your favourite editor :: + + /etc/libvirt/virtqemud.conf + /etc/libvirt/libvirtd.conf + $XDG_CONFIG_HOME/libvirt/libvirtd.conf + $XDG_CONFIG_HOME/libvirt/virtqemud.conf + +- find & replace, or set the apropriate `Log outputs`_ and `Log filters`_, e.g :: + + log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" + log_outputs="1:file:/var/log/libvirt/libvirtd.log" + +- save and exit +- restart the corresponding service/daemon e.g. :: + + systemctl restart virtqemud.socket + systemctl restart libvirtd.socket + systemctl restart libvirtd.service *Note:* Libvirt prior to the ``libvirt-4.4.0`` release didn't support globbing patterns and thus requires more configuration. See `Legacy (pre-4.4.0) libvirt daemon logging configuration`_. -However, when you are using the session mode ``qemu:///session`` or you run the -``libvirtd`` as unprivileged user you will find configuration file under -``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. - Runtime setting --------------- @@ -127,13 +176,16 @@ after the daemon restarts, since the new session can make the anomaly "disappear". Therefore, it's possible to enable the debug logs during runtime using libvirt administration API. To use it conveniently, there's the ``virt-admin`` client provided by the ``libvirt-admin`` package. Use the -package manager provided by your distribution to install this package. Once you -have it installed, run the following as root to see the set of log filters -currently being active: +package manager provided by your distribution to install this package. + +**Important**: Substitute ``virt-admin -c $ADMIN_URI`` per above guidance in +place of ``virt-admin`` in the examples below if needed. + +The following command allows to query the list of log filters currently active: :: - # virt-admin daemon-log-filters + # virt-admin daemon-log-filters Logging filters: 3:remote 4:util.json 4:rpc In order to change this set, run the same command as root, this time with your -- 2.34.1
On Mon, Jan 17, 2022 at 04:39:16PM +0100, Peter Krempa wrote:
Modular daemons are now the default in many new installations, thus we need to include steps how to determine that modular daemons are used and modular-daemon specific locations for the config files and admin URIs.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase/debuglogs.rst | 94 +++++++++++++++++++++++++++++++--------- 1 file changed, 73 insertions(+), 21 deletions(-)
diff --git a/docs/kbase/debuglogs.rst b/docs/kbase/debuglogs.rst index fc75ef14a4..ea7c2c1d52 100644 --- a/docs/kbase/debuglogs.rst +++ b/docs/kbase/debuglogs.rst @@ -87,38 +87,87 @@ more information than lack something important. Libvirt daemons logging configuration =====================================
-Persistent setting ------------------- +Libvirt daemons can be configured either via a config file or via the +administration API. The configuration location depends on multiple factors.
-The daemon configuration files location is dependent on `connection -URI <https://libvirt.org/uri.html>`__. For ``qemu:///system``: +Session vs. system daemons +--------------------------
s/\.//
+The libvirt daemons run either on ``system`` level or on ``session`` (user)
s/^The// s/on/in the/g s/level/mode/g
+level, depending on the configuration of the host and the available permission +levels.
s/the available/available/
+The `connection URI <https://libvirt.org/uri.html>`__ influences which daemon +the client is communicating with.
s/is communicating/will communicate with "A few facts about the two runtime modes"
-- open ``/etc/libvirt/libvirtd.conf`` in your favourite editor -- find & replace, or set these variables: +``system`` daemon e.g ``qemu:///system`` URI
-:: + * config files are usually placed in ``/etc/libvirt/``
- log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" - log_outputs="1:file:/var/log/libvirt/libvirtd.log" + * ``virt-admin`` URI ending in ``/system`` should be used.
-- save and exit -- restart libvirtd service
There should be a subsubsection for both IMO, how about: "System mode - all connection URIs will end with '/system', e.g. ``qemu:///system``` - config files are usually placed..." I deliberately left out the virt-admin part, because you're mentioning it again in 'Modular vs monolithic daemons', it's enough if virt-admin is mentioned once only. ...analogically, the same apples to the 'session' mode section below...
+``session`` daemon e.g. ``qemu:///session`` URI
-:: + * config files are usually placed in ``$XDG_CONFIG_HOME/libvirt/`` directory + + * ``virt-admin`` URI ending in ``/session`` should be used + +Modular vs. monolithic daemons +------------------------------ + +While the config file name for the monolithic daemon is simple, in case when +modular daemons are used, there's a lot of options. + +All libvirt daemons support the logging configuration per this document, but +in most cases it's required to configure logging for the appropriate.
How about: "While there is only a single 'libvirtd.conf' configuration file in case of the monolithic daemon setup, each of the modular daemons has their own configuration file giving you a lot of possibilities how to configure them individually including logging. Realistically though, logging will have to be configured only for a single or a couple of daemons in case debug logs are requested."
+ +Refer to `documentation about daemons <../daemons.html#checking-whether-modular-monolithic-mode-is-in-use>`__ +to figure out which is in use by your system. + + Modular daemons
Same as with system vs session --> should be a subsubsection
+ + The configuration of modular daemons is in file named after the daemon. E.g. + for ``qemu:///system`` connection this is the ``virtqemud`` daemon and + correspondingly:
How about: "Configuration files in the modular daemons setup are named after individual daemons, each prefixed with 'virt', e.g. in case of the ``qemu:///system`` hypervisor connection, the underlying daemon handling it is called ``virtqemud`` and so the configuration file will be named ``virtqemud.conf``. For runtime configuration with ``virt-admin`` (see below) ``virtqemud:///system`` or ``virtqemud:///session`` URIs would be used depending on which runtime mode is used for the daemons (see above)." ...analogically, you can use the same with a few changes for the monolithic subsection...
+ + * ``virtqemud.conf`` config file is used + + * ``virtqemud:///system`` or ``virtqemud:///session`` admin URI is used
- systemctl restart libvirtd.service + Monolithic daemon + + * ``libvirtd.conf`` config file is used + + * ``libvirtd:///system`` or ``libvirtd:///session`` admin URI is used + when the modular qemu hypervisor driver ``virtqemud`` + +Persistent setting +------------------ + "In order to setup libvirt logging persistently, follow the steps below:"
+- open the appropriate daemon config file in your favourite editor :: + + /etc/libvirt/virtqemud.conf + /etc/libvirt/libvirtd.conf + $XDG_CONFIG_HOME/libvirt/libvirtd.conf + $XDG_CONFIG_HOME/libvirt/virtqemud.conf + +- find & replace, or set the apropriate `Log outputs`_ and `Log filters`_, e.g :: + + log_filters="3:remote 4:event 3:util.json 3:rpc 1:*" + log_outputs="1:file:/var/log/libvirt/libvirtd.log" + +- save and exit +- restart the corresponding service/daemon e.g. :: + + systemctl restart virtqemud.socket + systemctl restart libvirtd.socket + systemctl restart libvirtd.service
*Note:* Libvirt prior to the ``libvirt-4.4.0`` release didn't support globbing patterns and thus requires more configuration. See `Legacy (pre-4.4.0) libvirt daemon logging configuration`_.
-However, when you are using the session mode ``qemu:///session`` or you run the -``libvirtd`` as unprivileged user you will find configuration file under -``$XDG_CONFIG_HOME/libvirt/libvirtd.conf``. - Runtime setting ---------------
@@ -127,13 +176,16 @@ after the daemon restarts, since the new session can make the anomaly "disappear". Therefore, it's possible to enable the debug logs during runtime using libvirt administration API. To use it conveniently, there's the ``virt-admin`` client provided by the ``libvirt-admin`` package. Use the -package manager provided by your distribution to install this package. Once you -have it installed, run the following as root to see the set of log filters -currently being active: +package manager provided by your distribution to install this package. + +**Important**: Substitute ``virt-admin -c $ADMIN_URI`` per above guidance in
s/per above/according to the guideline in the sections above
+place of ``virt-admin`` in the examples below if needed. + +The following command allows to query the list of log filters currently active:
"...list of currently active log filters:" Erik
On Mon, Jan 17, 2022 at 16:39:08 +0100, Peter Krempa wrote:
Modular daemons are the default in many cases yet, one of our most frequented documents neglects to describe the intricacies of the setup.
Browsable result is available at: https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1983126893/artifacts/website/daem... https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1983126893/artifacts/website/kbas...
Oops, I've nuked the CI pipeline results so these are broken, here's a new run: https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1987326040/artifacts/website/daem... https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1987326040/artifacts/website/kbas...
On Mon, Jan 17, 2022 at 04:39:08PM +0100, Peter Krempa wrote:
Modular daemons are the default in many cases yet, one of our most frequented documents neglects to describe the intricacies of the setup.
Browsable result is available at: https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1983126893/artifacts/website/daem... https://pipo.sk.gitlab.io/-/libvirt/-/jobs/1983126893/artifacts/website/kbas...
Sorry for TLS warnings. Gitlab didn't really think through allowing users to pick username with a dot and I refuse to change it.
I had more stylistic nitpicks in 8/8, but overall it looks good. Thank you for improving our knowledge base documentation. Erik
participants (2)
-
Erik Skultety -
Peter Krempa