[libvirt] [PATCH] 0/8 logging infrastructure for libvirt[d]

the following set of patches implement the logging infrastructure based on the discussion about it last month. It is based on the following interfaces, and in a large part reflects the log4j principles. 4 level of logging priorities: typedef enum { VIR_LOG_DEBUG = 1, VIR_LOG_INFO, VIR_LOG_WARN, VIR_LOG_ERROR, } virLogPriority; An output is a channel to output logging informations and a filter is a rule to allow or not log information to flow through. At any given time a set of output and filters can be defined though a public API (but I didn't put them yet in libvirt.h that's still confined in logging.h ATM) /** * virLogOutputFunc: * @data: extra output logging data * @category: the category for the message * @priority: the priority for the message * @msg: the message to log, preformatted and zero terminated * @len: the lenght of the message in bytes without the terminating zero * * Callback function used to output messages * * Returns the number of bytes written or -1 in case of error */ typedef int (*virLogOutputFunc) (void *data, const char *category, int priority, const char *str, int len); /** * virLogCloseFunc: * @data: extra output logging data * * Callback function used to close a log output */ typedef void (*virLogCloseFunc) (void *data); extern int virLogSetDefaultPriority(int priority); extern int virLogDefineFilter(const char *match, int priority, int flags); extern int virLogDefineOutput(virLogOutputFunc f, virLogCloseFunc c, void *data, int priority, int flags); The default priority allows to set a default level of logging, individual logs matching filters will follow the rules defined for the filters, i.e. if the logged data is of a lower level than what the first filter matching it then it is dropped. The internal APIs defines a function to log a message: extern void virLogMessage(const char *category, int priority, int flags, const char *fmt, ...); the category is in general the file name as defined by __FILE__ The matching is done against the category based on a substring, for example "xen" would match all logs emitted by xend_internal.c xen_inotify.c xen_internal.c xen_unified.c Something more complex could be defined for example matching based on regexp or based on the full message content and not just the category but this gets heavier and could be done later using specific flags in virLogDefineFilter. The preferred user interface for setting the output and filters is through environment variables for application linking to libvirt LIBVIRT_DEBUG defines the default logging level from 1 to 4 LIBVIRT_LOG_FILTERS defines the set filters LIBVIRT_LOG_OUTPUTS defines the set outputs and though the configuration file libvirtd.conf for the daemon with the corresponding values: log_level log_filters log_outputs The convenience internal functions virLogParseFilters and virLogParseOutputs parse the format of the values for filters and outputs, so they share the same format explained in libvirtd.conf: ------------------------------------------------------------------ # Logging filters: # A filter allows to select a different logging level for a given category # of logs # The format for a filter is: # x:name # where name is a match string e.g. remote or qemu # the x prefix is the minimal level where matching messages should be logged # 1: DEBUG # 2: INFO # 3: WARNING # 4: ERROR # # Multiple filter can be defined in a single @filters, they just need to be # separated by spaces. # # e.g: # log_filters="3:remote 4:event" # to only get warning or errors from the remote layer and only errors from # the event layer. # Logging outputs: # An output is one of the places to save logging informations # The format for an output can be: # x:stderr # output goes to stderr # x:syslog:name # use syslog for the output and use the given name as the ident # x:file:file_path # output to a file, with the given filepath # In all case the x prefix is the minimal level, acting as a filter # 0: everything # 1: DEBUG # 2: INFO # 3: WARNING # 4: ERROR # # Multiple output can be defined , they just need to be separated by spaces. # e.g.: # log_outputs="3:syslog:libvirtd" # to log all warnings and errors to syslog under the libvirtd ident ----------------------------------------------------------------------- In practice export LIBVIRT_DEBUG=1 export LIBVIRT_LOG_OUTPUTS="0:file:virsh.log" and then running virsh will accumulate all logging to the virsh.log file in the current directory. One thing which I feel is somewhat incomplete is that it's impossible to remotely get debugging output from the libvirt daemon serving the requests. Currently all logs are also accumulated in a cyclic logging buffer, I would associate a dump function later to be hooked for example to a signal handler in the daemon. But I'm unsure we should allow dumping logging information to the remote end, probably not the whole set. So there is still room for a bit more development, but I think it's usable as-is and already cleans quite a bit of the various logging interface scattered in the various modules. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

On Wed, Dec 17, 2008 at 04:06:38PM +0100, Daniel Veillard wrote:
In practice export LIBVIRT_DEBUG=1 export LIBVIRT_LOG_OUTPUTS="0:file:virsh.log" and then running virsh will accumulate all logging to the virsh.log file in the current directory.
This looks great.
One thing which I feel is somewhat incomplete is that it's impossible to remotely get debugging output from the libvirt daemon serving the requests. Currently all logs are also accumulated in a cyclic logging buffer, I would associate a dump function later to be hooked for example to a signal handler in the daemon. But I'm unsure we should allow dumping logging information to the remote end, probably not the whole set.
I've been wondering whether we should create an explicit tool for admin of the libvirtd daemon itself. Basically something for querying state, and performing operations wrt to the daemon for things outside the scope of the libvirt API. For example, something to get details of active client connections, and forceably drop a client. Being able to have an API to configure the log level / setup of the dameon would be a useful thing. We can easily layer this kind of thing in as another RPC protocol in parallel with the existing protocol, and have simple CLI tool libvirtd-admin. SQUID has a 'squidclient' tool and a CGI script for administration of the server itself. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Daniel P. Berrange wrote:
On Wed, Dec 17, 2008 at 04:06:38PM +0100, Daniel Veillard wrote:
In practice export LIBVIRT_DEBUG=1 export LIBVIRT_LOG_OUTPUTS="0:file:virsh.log" and then running virsh will accumulate all logging to the virsh.log file in the current directory.
This looks great.
One thing which I feel is somewhat incomplete is that it's impossible to remotely get debugging output from the libvirt daemon serving the requests. Currently all logs are also accumulated in a cyclic logging buffer, I would associate a dump function later to be hooked for example to a signal handler in the daemon. But I'm unsure we should allow dumping logging information to the remote end, probably not the whole set.
I've been wondering whether we should create an explicit tool for admin of the libvirtd daemon itself. Basically something for querying state, and performing operations wrt to the daemon for things outside the scope of the libvirt API. For example, something to get details of active client connections, and forceably drop a client. Being able to have an API to configure the log level / setup of the dameon would be a useful thing. We can easily layer this kind of thing in as another RPC protocol in parallel with the existing protocol, and have simple CLI tool libvirtd-admin. SQUID has a 'squidclient' tool and a CGI script for administration of the server itself.
Daniel
I think a daemon control interface is a great idea. Being able to change things like loglevel without restarting the daemon is a real plus for administrators. Dave

On Wed, Dec 17, 2008 at 03:57:37PM +0000, Daniel P. Berrange wrote:
On Wed, Dec 17, 2008 at 04:06:38PM +0100, Daniel Veillard wrote:
In practice export LIBVIRT_DEBUG=1 export LIBVIRT_LOG_OUTPUTS="0:file:virsh.log" and then running virsh will accumulate all logging to the virsh.log file in the current directory.
This looks great.
Okay, I have now commited the updated patches, there is still one augeas file I need to update, some test to be fixed because the output is slightly different now and increase documentation.
One thing which I feel is somewhat incomplete is that it's impossible to remotely get debugging output from the libvirt daemon serving the requests. Currently all logs are also accumulated in a cyclic logging buffer, I would associate a dump function later to be hooked for example to a signal handler in the daemon. But I'm unsure we should allow dumping logging information to the remote end, probably not the whole set.
I've been wondering whether we should create an explicit tool for admin of the libvirtd daemon itself. Basically something for querying state, and performing operations wrt to the daemon for things outside the scope of the libvirt API. For example, something to get details of active client connections, and forceably drop a client. Being able to have an API to configure the log level / setup of the dameon would be a useful thing. We can easily layer this kind of thing in as another RPC protocol in parallel with the existing protocol, and have simple CLI tool libvirtd-admin. SQUID has a 'squidclient' tool and a CGI script for administration of the server itself.
You envision that configuration tool to be able to work remotely ? Why not make that part of libvirt API, basically it's not that different from current Node operations, or maybe isolate them in a separate include header... Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

On Mon, Dec 22, 2008 at 02:01:38PM +0100, Daniel Veillard wrote:
On Wed, Dec 17, 2008 at 03:57:37PM +0000, Daniel P. Berrange wrote:
On Wed, Dec 17, 2008 at 04:06:38PM +0100, Daniel Veillard wrote:
In practice export LIBVIRT_DEBUG=1 export LIBVIRT_LOG_OUTPUTS="0:file:virsh.log" and then running virsh will accumulate all logging to the virsh.log file in the current directory.
This looks great.
Okay, I have now commited the updated patches, there is still one augeas file I need to update, some test to be fixed because the output is slightly different now and increase documentation.
make check is fixed now, Just extra docs are needed, I will finish those tomorrow, Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

On Mon, Dec 22, 2008 at 05:17:22PM +0100, Daniel Veillard wrote:
On Mon, Dec 22, 2008 at 02:01:38PM +0100, Daniel Veillard wrote: make check is fixed now, Just extra docs are needed, I will finish those tomorrow,
Commited, available at http://libvirt.org/logging.html I have put them under Documentation > Deployment > Logging that was looking the most logical place to store those, Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
participants (3)
-
Daniel P. Berrange
-
Daniel Veillard
-
Dave Allan