9 a.m.
Hello,
## SETUP
CentOS 8 Stream
libguestfs-tools.noarch 1:1.40.2-24.el8.plesk
libvirt.x86_64 7.6.0-6.el8s
## ISSUE REPRODUCTION
lsof -p $(cat /run/libvirtd.pid) | wc -l # here we have N open file descriptors by
libvirtd
guestfish --ro -d domain1
run # here we have N+1 open file descriptors by libvirtd
Ctrl+D # and we still have N+1 open file descriptors by libvirtd
virt-df -d domain1 # after completion we still having N++ open file descriptors by
libvirtd
To fix this issue i am force to restart libvirtd
9:06 a.m.
On Tue, Feb 22, 2022 at 06:00:50PM +0300, Дмитрий wrote:
Hello,
## SETUP
CentOS 8 Stream
libguestfs-tools.noarch 1:1.40.2-24.el8.plesk
libvirt.x86_64 7.6.0-6.el8s
The current libvirt shipping in CentOS 8 Stream is
8.0.0-2.module_el8.6.0+1087+b42c8331
which is 6 months newer than what you have installed. So I'd recommend
ensuring you have latest CentOS stream content as a first step.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:23 a.m.
Updated libvirt with:
# dnf install @virt
Not its 2.module_el8.6.0+1087+b42c8331
Testing:
# virt-df -d kvm3551
But still, issue remains. Here how it looks like in /proc (all these socket/pipe symlinks
are broken):
# ls -lah /proc/$(pgrep -f libvirtd)/fd/*
lrwx------ 1 root root 64 фев 22 18:17 /proc/3600693/fd/49 ->
'socket:[99301766]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/5 ->
'socket:[99234506]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/51 ->
'socket:[99295594]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/52 ->
'socket:[99288645]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/53 ->
'socket:[99296279]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/54 ->
'socket:[99289772]'
lrwx------ 1 root root 64 фев 22 18:17 /proc/3600693/fd/55 ->
'socket:[99300744]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/56 ->
'socket:[99284626]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/57 ->
'socket:[99294777]'
lrwx------ 1 root root 64 фев 22 18:17 /proc/3600693/fd/58 ->
'socket:[99297768]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/59 ->
'socket:[99279867]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/6 ->
'socket:[99282298]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/60 ->
'socket:[99286467]'
l-wx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/7 -> /run/libvirtd.pid
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/70 ->
'socket:[99286794]'
lr-x------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/8 -> 'pipe:[99282299]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/81 ->
'socket:[99281902]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/82 ->
'socket:[99288059]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/84 ->
'socket:[99285702]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/85 ->
'socket:[99300385]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/88 ->
'socket:[99291046]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/89 ->
'socket:[99299347]'
l-wx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/9 -> 'pipe:[99282299]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/92 ->
'socket:[99284744]'
9:33 a.m.
On Tue, Feb 22, 2022 at 06:23:13PM +0300, Дмитрий wrote:
Updated libvirt with:
# dnf install @virt
Not its 2.module_el8.6.0+1087+b42c8331
Testing:
# virt-df -d kvm3551
But still, issue remains. Here how it looks like in /proc (all these socket/pipe symlinks
are broken):
# ls -lah /proc/$(pgrep -f libvirtd)/fd/*
lrwx------ 1 root root 64 фев 22 18:17 /proc/3600693/fd/49 ->
'socket:[99301766]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/5 ->
'socket:[99234506]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/51 ->
'socket:[99295594]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/52 ->
'socket:[99288645]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/53 ->
'socket:[99296279]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/54 ->
'socket:[99289772]'
lrwx------ 1 root root 64 фев 22 18:17 /proc/3600693/fd/55 ->
'socket:[99300744]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/56 ->
'socket:[99284626]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/57 ->
'socket:[99294777]'
lrwx------ 1 root root 64 фев 22 18:17 /proc/3600693/fd/58 ->
'socket:[99297768]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/59 ->
'socket:[99279867]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/6 ->
'socket:[99282298]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/60 ->
'socket:[99286467]'
l-wx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/7 -> /run/libvirtd.pid
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/70 ->
'socket:[99286794]'
lr-x------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/8 ->
'pipe:[99282299]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/81 ->
'socket:[99281902]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/82 ->
'socket:[99288059]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/84 ->
'socket:[99285702]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/85 ->
'socket:[99300385]'
lrwx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/88 ->
'socket:[99291046]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/89 ->
'socket:[99299347]'
l-wx------ 1 root root 64 фев 22 18:13 /proc/3600693/fd/9 ->
'pipe:[99282299]'
lrwx------ 1 root root 64 фев 22 18:15 /proc/3600693/fd/92 ->
'socket:[99284744]'
It is generally more useful to use 'lsof -p $LIBVIRT_PID', so you
get readable names for sockets.
Also compare before & after so you can identify which is the new
"leaked" file.
Also is that a one time leak, or is a new FD leaked for every
virt-df command that is run ?
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:45 a.m.
On 2/22/22 16:36, Дмитрий wrote:
# lsof -p $(cat /run/libvirtd.pid) > run1
# virt-df -d kvm3586
# lsof -p $(cat /run/libvirtd.pid) > run2
# diff run1 run2
196a197
> libvirtd 3618992 root 54u unix 0xffff902dc0c58440 0t0 99330533
type=STREAM
I wonder whether this is the glib event loop issue I've fixed a while
ago. Symptoms look similar.
https://gitlab.com/libvirt/libvirt/-/commit/5de203f8795d96229d2663e9ea1a2...
Let me check if that's backported into CentOS and what versions.
Michal
5:40 a.m.
Any update on this issue? Does anyone have a success on reproducing that? Should I
upgrade/downgrade/sidegrade?
Additional info:
I am using CentOS 8 Stream, but with kernel
5.14.15-1.el8.elrepo.x86_64 #1 SMP Tue Oct 26 11:45:20 EDT 2021 x86_64 x86_64 x86_64
GNU/Linux
6:12 a.m.
On 3/1/22 12:40, Дмитрий wrote:
Any update on this issue? Does anyone have a success on reproducing
that? Should I upgrade/downgrade/sidegrade?
Additional info:
I am using CentOS 8 Stream, but with kernel
5.14.15-1.el8.elrepo.x86_64 #1 SMP Tue Oct 26 11:45:20 EDT 2021 x86_64 x86_64 x86_64
GNU/Linux
I believe this was fixed in RHEL-8, by a RHEL-only patch, because in
there glib backported a commit that broke our upstream workaround. And
since CentOS 8 Stream is behind RHEL you will get the fix eventually
(sorry I don't know the schedule). However, I don't think that the
RHEL-only patch can be merged upstream, because then it would break
distros with older glib that did not mangle it.
I'm sorry, I don't have a better answer.
Michal
878
days inactive
885
days old
7 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Michal Prívozník -
Дмитрий