On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote:
пн, 7 февр. 2022 г. в 13:29, Michal Prívozník
<mprivozn(a)redhat.com>:
> On 2/7/22 09:22, Nikolay Shirokovskiy wrote:
> > Hi, all.
> >
> > Libvirt QEMU driver writes QEMU process log to
> > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by libvirt
> > that is no API client knows about it and no client will remove it. Thus
> > this file stays forever. I guess it does not cause harddisk space waste
> > in some deploys as the number of VMs thru node lifecycle is not very
> large.
> >
> > In Virtuozzo we have a service which checks VMs disks every 5 min using
> > guestfs. This generates about 100k files and about 400MiB disk usage per
> > VM per year. This is a lot. I guess we could refine the service to avoid
> > this issue yet I think the issue is general and need to be addressed.
> >
> > If this is agreed then in terms of architecture should we have a
> > distinct timer/service to cleanup log files or we'd better clean up in
> > scope of qemu driver itself?
>
> I believe that this was one the issues that virtlogd tried to solve. By
> doing log rotation and keeping the last three files (by default). This
> can be fine tuned in virtlogd.conf.
>
Hi, Michal.
Yes virtlogd/logrotate can limit the size of logs for a single VM. But if
we keep logs forever for every VM that existed at some point in time then
the total size of logs can increase without any limit.
I should mention that when we check VMs disks using guestfs it creates a
new transient VM every time. That is why we have so many logs left. But
most of these logs are not of interest - a VM was deleted a long time ago.
So the suggestion is to keep logs for deleted VMs only for some period of
time, say a month.
We have a logrotate config, that was pre-dating the existance of
virtlogd, so its rules currently should be a no-op because
virtlogd will have already rotated stuff before the logrotrate
rules trigger.
I wonder, however, if there is a way to set the logrotate rules
to delete files, without interfering/clashing with virtlogd
rollover.
A completely different, perhaps complementary, approach would be
to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS
that libguestfs could pass, on the basis that having these logs
persist is essentially a waste of time for transient VMs like
these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if
asked to run in a debugging mode perhaps.
Regards,
Daniel
--
|:
https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|:
https://libvirt.org -o-
https://fstop138.berrange.com :|
|:
https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|