1:24 p.m.
With logrotates copytruncate when e.g. domain1 doesn't exist anymore
/var/log/libvirt/qemu/domain1.log will still exist after rotation even
though it will never be written to. When new domain names keep getting
used this leads to a lot of empty logfiles. This may lead to slowdown or
lack of free disk space / inodes.
Fix this by replacing copytruncate with the apropriate postrotate
command to reopen log files. Thus after the apropriate time log files
for deleted domains will be gone. This also has the advantage that the
chance for loss of a few lines during copytruncate is gone.
Signed-off-by: Jan Zerebecki <jan.suse(a)zerebecki.de>
---
src/remote/libvirtd.libxl.logrotate.in | 4 +++-
src/remote/libvirtd.logrotate.in | 4 +++-
src/remote/libvirtd.lxc.logrotate.in | 4 +++-
src/remote/libvirtd.qemu.logrotate.in | 4 +++-
4 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/src/remote/libvirtd.libxl.logrotate.in
b/src/remote/libvirtd.libxl.logrotate.in
index cb7f07d846..d4f25f8cab 100644
--- a/src/remote/libvirtd.libxl.logrotate.in
+++ b/src/remote/libvirtd.libxl.logrotate.in
@@ -4,5 +4,7 @@
rotate 4
compress
delaycompress
- copytruncate
+ postrotate
+ /usr/bin/killall -USR1 virtlogd
+ endscript
}
diff --git a/src/remote/libvirtd.logrotate.in b/src/remote/libvirtd.logrotate.in
index 4e02510c8b..73dab7579f 100644
--- a/src/remote/libvirtd.logrotate.in
+++ b/src/remote/libvirtd.logrotate.in
@@ -4,6 +4,8 @@
rotate 4
compress
delaycompress
- copytruncate
minsize 100k
+ postrotate
+ /usr/bin/killall -USR1 virtlogd
+ endscript
}
diff --git a/src/remote/libvirtd.lxc.logrotate.in b/src/remote/libvirtd.lxc.logrotate.in
index 2bb9dfba12..eed47bb55f 100644
--- a/src/remote/libvirtd.lxc.logrotate.in
+++ b/src/remote/libvirtd.lxc.logrotate.in
@@ -4,5 +4,7 @@
rotate 4
compress
delaycompress
- copytruncate
+ postrotate
+ /usr/bin/killall -USR1 virtlogd
+ endscript
}
diff --git a/src/remote/libvirtd.qemu.logrotate.in
b/src/remote/libvirtd.qemu.logrotate.in
index cdb399ef23..95407cec1a 100644
--- a/src/remote/libvirtd.qemu.logrotate.in
+++ b/src/remote/libvirtd.qemu.logrotate.in
@@ -4,5 +4,7 @@
rotate 4
compress
delaycompress
- copytruncate
+ postrotate
+ /usr/bin/killall -USR1 virtlogd
+ endscript
}
--
2.20.1
1:03 a.m.
On Tue, Jul 02, 2019 at 20:24:09 +0200, Jan Zerebecki wrote:
With logrotates copytruncate when e.g. domain1 doesn't exist
anymore
/var/log/libvirt/qemu/domain1.log will still exist after rotation even
though it will never be written to. When new domain names keep getting
used this leads to a lot of empty logfiles. This may lead to slowdown or
lack of free disk space / inodes.
Fix this by replacing copytruncate with the apropriate postrotate
command to reopen log files. Thus after the apropriate time log files
for deleted domains will be gone. This also has the advantage that the
chance for loss of a few lines during copytruncate is gone.
Signed-off-by: Jan Zerebecki <jan.suse(a)zerebecki.de>
---
src/remote/libvirtd.libxl.logrotate.in | 4 +++-
src/remote/libvirtd.logrotate.in | 4 +++-
src/remote/libvirtd.lxc.logrotate.in | 4 +++-
src/remote/libvirtd.qemu.logrotate.in | 4 +++-
4 files changed, 12 insertions(+), 4 deletions(-)
I'm not sure that this is the right thing to do. virtlogd has some
internal log rotation mechanisms and also the SIGUSR1 action is reserved
for re-exec updates and not for dealing with modified files
Additionally the internals are supposed to keep all the FD's open during
re-exec so I'm not even sure that this will help.
1:32 a.m.
On 03/07/2019 08.03, Peter Krempa wrote:
I'm not sure that this is the right thing to do. virtlogd has
some
internal log rotation mechanisms
logrotate is already in use here and this patch doesn't change what is
rotated nor how often it is rotated.
and also the SIGUSR1 action is reserved
for re-exec updates and not for dealing with modified files
The mechanism (re-exec) and implementation details for both uses do not
conflict.
Additionally the internals are supposed to keep all the FD's open
during
re-exec so I'm not even sure that this will help.
This test shows that it helps:
# ls -la /var/log/libvirt/qemu/foo.log
-rw------- 1 root root 0 Jun 30 00:00 /var/log/libvirt/qemu/foo.log
# mv /var/log/libvirt/qemu/foo.log /var/log/libvirt/qemu/foo.log.0
# /usr/bin/killall -USR1 virtlogd
# ls -la /var/log/libvirt/qemu/foo.log
-rw------- 1 root root 0 Jul 3 08:06 /var/log/libvirt/qemu/foo.log
--
Best regards,
Jan Zerebecki
1:55 a.m.
On Wed, Jul 03, 2019 at 08:32:21 +0200, Jan Zerebecki wrote:
On 03/07/2019 08.03, Peter Krempa wrote:
> I'm not sure that this is the right thing to do. virtlogd has some
> internal log rotation mechanisms
logrotate is already in use here and this patch doesn't change what is
rotated nor how often it is rotated.
> and also the SIGUSR1 action is reserved
> for re-exec updates and not for dealing with modified files
The mechanism (re-exec) and implementation details for both uses do not
conflict.
The problem is that the logrotate configs predate use of virtlogd and I
don't think they were adapted to be used together. In fact if I remember
correctly the idea of virtlogd was to avoid using logrotate altogether.
Additionally virtlogd is currently used only with the qemu VM log files.
Everything else logs itself, so your changes to libvirtd/lxc and others
seem to be unnecessary.
2:27 a.m.
On 03/07/2019 08.55, Peter Krempa wrote:
On Wed, Jul 03, 2019 at 08:32:21 +0200, Jan Zerebecki wrote:
> On 03/07/2019 08.03, Peter Krempa wrote:
>> I'm not sure that this is the right thing to do. virtlogd has some
>> internal log rotation mechanisms
>
> logrotate is already in use here and this patch doesn't change what is
> rotated nor how often it is rotated.
>
>> and also the SIGUSR1 action is reserved
>> for re-exec updates and not for dealing with modified files
>
> The mechanism (re-exec) and implementation details for both uses do not
> conflict.
The problem is that the logrotate configs predate use of virtlogd and I
don't think they were adapted to be used together. In fact if I remember
correctly the idea of virtlogd was to avoid using logrotate altogether.
virtlogd rotates by size and logrotate in this config by time, so they
work fine together. Yes, you could implement rotating out logfiles from
deleted domains in virtlogd, but I don't see a reason to reimplement it.
Additionally virtlogd is currently used only with the qemu VM log
files.
Everything else logs itself, so your changes to libvirtd/lxc and others
seem to be unnecessary.
I don't have some of these handy to test, how do you get them to reopen
logs?
--
Best regards,
Jan Zerebecki
3:42 a.m.
On Wed, Jul 03, 2019 at 08:55:15AM +0200, Peter Krempa wrote:
On Wed, Jul 03, 2019 at 08:32:21 +0200, Jan Zerebecki wrote:
> On 03/07/2019 08.03, Peter Krempa wrote:
> > I'm not sure that this is the right thing to do. virtlogd has some
> > internal log rotation mechanisms
>
> logrotate is already in use here and this patch doesn't change what is
> rotated nor how often it is rotated.
>
> > and also the SIGUSR1 action is reserved
> > for re-exec updates and not for dealing with modified files
>
> The mechanism (re-exec) and implementation details for both uses do not
> conflict.
The problem is that the logrotate configs predate use of virtlogd and I
don't think they were adapted to be used together. In fact if I remember
correctly the idea of virtlogd was to avoid using logrotate altogether.
Yes, logrotate should not be in effect when virtlogd is running. We should
ensure that logrotate rollover size is *larger* than the rollover size
configured for virtlogd. This would ensure virtlogd rolls over first, and
thus when logrotate runs it should see nothing todo.
Additionally virtlogd is currently used only with the qemu VM log
files.
Everything else logs itself, so your changes to libvirtd/lxc and others
seem to be unnecessary.
Yep, they're simply wrong.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
10:26 a.m.
On 7/3/19 2:42 AM, Daniel P. Berrangé wrote:
On Wed, Jul 03, 2019 at 08:55:15AM +0200, Peter Krempa wrote:
> On Wed, Jul 03, 2019 at 08:32:21 +0200, Jan Zerebecki wrote:
>> On 03/07/2019 08.03, Peter Krempa wrote:
>>> I'm not sure that this is the right thing to do. virtlogd has some
>>> internal log rotation mechanisms
>>
>> logrotate is already in use here and this patch doesn't change what is
>> rotated nor how often it is rotated.
>>
>>> and also the SIGUSR1 action is reserved
>>> for re-exec updates and not for dealing with modified files
>>
>> The mechanism (re-exec) and implementation details for both uses do not
>> conflict.
>
> The problem is that the logrotate configs predate use of virtlogd and I
> don't think they were adapted to be used together. In fact if I remember
> correctly the idea of virtlogd was to avoid using logrotate altogether.
Yes, logrotate should not be in effect when virtlogd is running. We should
ensure that logrotate rollover size is *larger* than the rollover size
configured for virtlogd. This would ensure virtlogd rolls over first, and
thus when logrotate runs it should see nothing todo.
Hmm, it sounds like the proper fix is to simply remove
src/remote/libvirtd.qemu.logrotate.in?
Regards,
Jim
10:56 a.m.
On Wed, Jul 03, 2019 at 03:26:36PM +0000, Jim Fehlig wrote:
On 7/3/19 2:42 AM, Daniel P. Berrangé wrote:
> On Wed, Jul 03, 2019 at 08:55:15AM +0200, Peter Krempa wrote:
>> On Wed, Jul 03, 2019 at 08:32:21 +0200, Jan Zerebecki wrote:
>>> On 03/07/2019 08.03, Peter Krempa wrote:
>>>> I'm not sure that this is the right thing to do. virtlogd has some
>>>> internal log rotation mechanisms
>>>
>>> logrotate is already in use here and this patch doesn't change what is
>>> rotated nor how often it is rotated.
>>>
>>>> and also the SIGUSR1 action is reserved
>>>> for re-exec updates and not for dealing with modified files
>>>
>>> The mechanism (re-exec) and implementation details for both uses do not
>>> conflict.
>>
>> The problem is that the logrotate configs predate use of virtlogd and I
>> don't think they were adapted to be used together. In fact if I remember
>> correctly the idea of virtlogd was to avoid using logrotate altogether.
>
> Yes, logrotate should not be in effect when virtlogd is running. We should
> ensure that logrotate rollover size is *larger* than the rollover size
> configured for virtlogd. This would ensure virtlogd rolls over first, and
> thus when logrotate runs it should see nothing todo.
Hmm, it sounds like the proper fix is to simply remove
src/remote/libvirtd.qemu.logrotate.in?
We didn't want to remove it because whether or not to use virtlogd is
a config file tunable. Hence we want to make sure that its default
settings end up being a no-op if virtlogd has already done its own
rollover
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
2017
days inactive
2018
days old
7 comments
4 participants
participants (4)
-
Daniel P. Berrangé -
Jan Zerebecki -
Jim Fehlig -
Peter Krempa