On 2/7/22 09:22, Nikolay Shirokovskiy wrote: > Hi, all. > > Libvirt QEMU driver writes QEMU process log to > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by libvirt > that is no API client knows about it and no client will remove it. Thus > this file stays forever. I guess it does not cause harddisk space waste > in some deploys as the number of VMs thru node lifecycle is not very large. > > In Virtuozzo we have a service which checks VMs disks every 5 min using > guestfs. This generates about 100k files and about 400MiB disk usage per > VM per year. This is a lot. I guess we could refine the service to avoid > this issue yet I think the issue is general and need to be addressed. > > If this is agreed then in terms of architecture should we have a > distinct timer/service to cleanup log files or we'd better clean up in > scope of qemu driver itself? I believe that this was one the issues that virtlogd tried to solve. By doing log rotation and keeping the last three files (by default). This can be fine tuned in virtlogd.conf.

On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote: > пн, 7 февр. 2022 г. в 13:29, Michal Prívozník <mprivozn(a)redhat.com&gt;: > > > On 2/7/22 09:22, Nikolay Shirokovskiy wrote: > > > Hi, all. > > > > > > Libvirt QEMU driver writes QEMU process log to > > > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by libvirt > > > that is no API client knows about it and no client will remove it. Thus > > > this file stays forever. I guess it does not cause harddisk space waste > > > in some deploys as the number of VMs thru node lifecycle is not very > > large. > > > > > > In Virtuozzo we have a service which checks VMs disks every 5 min using > > > guestfs. This generates about 100k files and about 400MiB disk usage per > > > VM per year. This is a lot. I guess we could refine the service to avoid > > > this issue yet I think the issue is general and need to be addressed. > > > > > > If this is agreed then in terms of architecture should we have a > > > distinct timer/service to cleanup log files or we'd better clean up in > > > scope of qemu driver itself? > > > > I believe that this was one the issues that virtlogd tried to solve. By > > doing log rotation and keeping the last three files (by default). This > > can be fine tuned in virtlogd.conf. > > > > Hi, Michal. > > Yes virtlogd/logrotate can limit the size of logs for a single VM. But if > we keep logs forever for every VM that existed at some point in time then > the total size of logs can increase without any limit. > > I should mention that when we check VMs disks using guestfs it creates a > new transient VM every time. That is why we have so many logs left. But > most of these logs are not of interest - a VM was deleted a long time ago. > > So the suggestion is to keep logs for deleted VMs only for some period of > time, say a month. We have a logrotate config, that was pre-dating the existance of virtlogd, so its rules currently should be a no-op because virtlogd will have already rotated stuff before the logrotrate rules trigger. I wonder, however, if there is a way to set the logrotate rules to delete files, without interfering/clashing with virtlogd rollover.

A completely different, perhaps complementary, approach would be to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS that libguestfs could pass, on the basis that having these logs persist is essentially a waste of time for transient VMs like these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if asked to run in a debugging mode perhaps.