Re: removing VMs logs

On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote: > пн, 7 февр. 2022 г. в 13:29, Michal Prívozník <mprivozn(a)redhat.com&gt;: > > > On 2/7/22 09:22, Nikolay Shirokovskiy wrote: > > > Hi, all. > > > > > > Libvirt QEMU driver writes QEMU process log to > > > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by libvirt > > > that is no API client knows about it and no client will remove it. Thus > > > this file stays forever. I guess it does not cause harddisk space waste > > > in some deploys as the number of VMs thru node lifecycle is not very > > large. > > > > > > In Virtuozzo we have a service which checks VMs disks every 5 min using > > > guestfs. This generates about 100k files and about 400MiB disk usage per > > > VM per year. This is a lot. I guess we could refine the service to avoid > > > this issue yet I think the issue is general and need to be addressed. > > > > > > If this is agreed then in terms of architecture should we have a > > > distinct timer/service to cleanup log files or we'd better clean up in > > > scope of qemu driver itself? > > > > I believe that this was one the issues that virtlogd tried to solve. By > > doing log rotation and keeping the last three files (by default). This > > can be fine tuned in virtlogd.conf. > > > > Hi, Michal. > > Yes virtlogd/logrotate can limit the size of logs for a single VM. But if > we keep logs forever for every VM that existed at some point in time then > the total size of logs can increase without any limit. > > I should mention that when we check VMs disks using guestfs it creates a > new transient VM every time. That is why we have so many logs left. But > most of these logs are not of interest - a VM was deleted a long time ago. > > So the suggestion is to keep logs for deleted VMs only for some period of > time, say a month. We have a logrotate config, that was pre-dating the existance of virtlogd, so its rules currently should be a no-op because virtlogd will have already rotated stuff before the logrotrate rules trigger. I wonder, however, if there is a way to set the logrotate rules to delete files, without interfering/clashing with virtlogd rollover.

A completely different, perhaps complementary, approach would be to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS that libguestfs could pass, on the basis that having these logs persist is essentially a waste of time for transient VMs like these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if asked to run in a debugging mode perhaps.