ср, 9 февр. 2022 г. в 12:30, Daniel P. Berrangé <berrange@redhat.com>:
On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote:
> пн, 7 февр. 2022 г. в 13:29, Michal Prívozník <mprivozn@redhat.com>:
>
> > On 2/7/22 09:22, Nikolay Shirokovskiy wrote:
> > > Hi, all.
> > >
> > > Libvirt QEMU driver writes QEMU process log to
> > > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by libvirt
> > > that is no API client knows about it and no client will remove it. Thus
> > > this file stays forever. I guess it does not cause harddisk space waste
> > > in some deploys as the number of VMs thru node lifecycle is not very
> > large.
> > >
> > > In Virtuozzo we have a service which checks VMs disks every 5 min using
> > > guestfs. This generates about 100k files and about 400MiB disk usage per
> > > VM per year. This is a lot. I guess we could refine the service to avoid
> > > this issue yet I think the issue is general and need to be addressed.
> > >
> > > If this is agreed then in terms of architecture should we have a
> > > distinct timer/service to cleanup log files or we'd better clean up in
> > > scope of qemu driver itself?
> >
> > I believe that this was one the issues that virtlogd tried to solve. By
> > doing log rotation and keeping the last three files (by default). This
> > can be fine tuned in virtlogd.conf.
> >
>
> Hi, Michal.
>
> Yes virtlogd/logrotate can limit the size of logs for a single VM. But if
> we keep logs forever for every VM that existed at some point in time then
> the total size of logs can increase without any limit.
>
> I should mention that when we check VMs disks using guestfs it creates a
> new transient VM every time. That is why we have so many logs left. But
> most of these logs are not of interest - a VM was deleted a long time ago.
>
> So the suggestion is to keep logs for deleted VMs only for some period of
> time, say a month.

We have a logrotate config, that was pre-dating the existance of
virtlogd, so its rules currently should be a no-op because
virtlogd will have already rotated stuff before the logrotrate
rules trigger.

I wonder, however, if there is a way to set the logrotate rules
to delete files, without interfering/clashing with virtlogd
rollover.


I investigated this approach a bit. This piece of configuration looks
most promising:

    weekly
    maxsize 2097153
    # copytruncate is off
    nocreate
    maxage 30
    sharedscripts
    postrotate
        kill -HUP virtlogd
    endscript

We also need to support reopening log files on SIGHUP in virtlogd of course.

'maxage' allows you to remove outdated rotated files. 'nocreate' is also required
as otherwise logrotate will create foo.log again with a fresh timestamp and
eventually foo.log will never be deleted. We need 'maxsize' instead of 'size'
also. The thing is 'maxage' is only applied on rotation which will never happened
with 'size' if log is abandoned.

Unfortunately 'maxsize' allows weekly rotations which will clash with virtlogd
rotations. I also think of introducing 'forceage' option instead of weekly/maxsize
pair to ask logrotate to apply 'maxage' even without rotation. Still I think
in this way we can clash with virtlogd on managing log files.

I guess we can either make delete/rename in virtlogd/logrotate in a very specific
order to handle races or introduce file locks to cooperate nicely. But to me
it does not look the right way. It will complicate logrotate because it will
need to cooperate with another log files manager.

I would add a new timer/service to cleanup log files. The service can be a very
simple shell script. It will need to cooperate with virtlogd too and we
can use file locks on log files for this purpose.

 

A completely different, perhaps complementary, approach would be
to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS
that libguestfs could pass, on the basis that having these logs
persist is essentially a waste of time for transient VMs like
these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if
asked to run in a debugging mode perhaps.


Yes this will make the log directory much cleaner. Even if we cleanup log
files on time basis we will still have a lot of not very meaningful logs
from guestfs in the log directory without this part.

Nikolay