ср, 9 февр. 2022 г. в 12:30, Daniel P. Berrangé <berrange@redhat.com>:
пн, 7 февр. 2022 г. в 13:29, Michal Prívozník <mprivozn@redhat.com>:
On 2/7/22 09:22, Nikolay Shirokovskiy wrote:
Hi, all.
Libvirt QEMU driver writes QEMU process log to /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by
that is no API client knows about it and no client will remove it. Thus this file stays forever. I guess it does not cause harddisk space waste in some deploys as the number of VMs thru node lifecycle is not very large.
In Virtuozzo we have a service which checks VMs disks every 5 min using guestfs. This generates about 100k files and about 400MiB disk usage
On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote: libvirt per
VM per year. This is a lot. I guess we could refine the service to avoid this issue yet I think the issue is general and need to be addressed.
If this is agreed then in terms of architecture should we have a distinct timer/service to cleanup log files or we'd better clean up in scope of qemu driver itself?
I believe that this was one the issues that virtlogd tried to solve. By doing log rotation and keeping the last three files (by default). This can be fine tuned in virtlogd.conf.
Hi, Michal.
Yes virtlogd/logrotate can limit the size of logs for a single VM. But if we keep logs forever for every VM that existed at some point in time then the total size of logs can increase without any limit.
I should mention that when we check VMs disks using guestfs it creates a new transient VM every time. That is why we have so many logs left. But most of these logs are not of interest - a VM was deleted a long time ago.
So the suggestion is to keep logs for deleted VMs only for some period of time, say a month.
We have a logrotate config, that was pre-dating the existance of virtlogd, so its rules currently should be a no-op because virtlogd will have already rotated stuff before the logrotrate rules trigger.
I wonder, however, if there is a way to set the logrotate rules to delete files, without interfering/clashing with virtlogd rollover.
I investigated this approach a bit. This piece of configuration looks most promising: weekly maxsize 2097153 # copytruncate is off nocreate maxage 30 sharedscripts postrotate kill -HUP virtlogd endscript We also need to support reopening log files on SIGHUP in virtlogd of course. 'maxage' allows you to remove outdated rotated files. 'nocreate' is also required as otherwise logrotate will create foo.log again with a fresh timestamp and eventually foo.log will never be deleted. We need 'maxsize' instead of 'size' also. The thing is 'maxage' is only applied on rotation which will never happened with 'size' if log is abandoned. Unfortunately 'maxsize' allows weekly rotations which will clash with virtlogd rotations. I also think of introducing 'forceage' option instead of weekly/maxsize pair to ask logrotate to apply 'maxage' even without rotation. Still I think in this way we can clash with virtlogd on managing log files. I guess we can either make delete/rename in virtlogd/logrotate in a very specific order to handle races or introduce file locks to cooperate nicely. But to me it does not look the right way. It will complicate logrotate because it will need to cooperate with another log files manager. I would add a new timer/service to cleanup log files. The service can be a very simple shell script. It will need to cooperate with virtlogd too and we can use file locks on log files for this purpose.
A completely different, perhaps complementary, approach would be to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS that libguestfs could pass, on the basis that having these logs persist is essentially a waste of time for transient VMs like these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if asked to run in a debugging mode perhaps.
Yes this will make the log directory much cleaner. Even if we cleanup log files on time basis we will still have a lot of not very meaningful logs from guestfs in the log directory without this part. Nikolay