12:25 p.m.
On 01/29/2016 05:08 AM, Peter Steele wrote:
We have been researching stuck zombie processes in our libvirt lxc
containers. What we found was:
1) Each zombie’s parent was pid 1. init which symlinks to systemd.
2) In some cases, the zombies were launched by systemd, in others the
zombie was inherited.
3) While the child is in the zombie state, the parent process
(systemd) /proc/1/status shows no pending signals.
4) Attaching gdb to systemd, there was 1 thread and it was waiting in
write() and the file being written was /dev/console.
This write() to the console never returns. We operated under the
assumption that systemd's SIGCHLD handler sets a bit and a foreground
thread (the only thread) would see that child processes needed
reaping. While the single thread is stuck in write(), the reaping
never takes place.
So why is write() blocking? The answer seems to be that there is
nothing draining the console and eventually it blocks write() when its
buffers become full. When we attached to the container's console, the
buffer is cleared allowing systemd’s write() to return. The zombies
are then reaped and everything goes back to normal.
Our “solution” was more of a workaround. systemd was altered to log
errors/warnings/etc to /dev/null instead of /dev/console. This
prevented the problem, only in that the console buffer was unlikely to
get filled up since systemd generally is the only then that writes to
it. This is definitely a hack though.
This may be a bug in the libvirt container library (you can't expect
something to periodically connect to a container's console to empty it
out). We suspect there may also be a configuration issue in our
containers with regards to the console.
Has anyone else observed this problem?
As I mentioned here, I think this may have to do with incorrect
container configuration with regards to the console. Much of the process
though is automated by libvirt itself so I'm not sure what I might be
missing. When a container is created, the xml config has this entry defined:
<console type='pty'>
<target type='lxc' port='0'/>
</console>
After starting the container, the console config in the xml changes, e.g.:
<console type='pty' tty='/dev/pts/2'>
<source path='/dev/pts/2'/>
<target type='lxc' port='0'/>
<alias name='console0'/>
</console>
In addition to these changes, a new entry is created under /dev/pts:
# ll /dev/pts
crw--w---- 1 root tty 136, 0 Jan 29 08:27 0
crw--w---- 1 root tty 136, 1 Jan 29 08:26 1
crw--w---- 1 root tty 136, 2 Jan 29 09:19 2 <---
crw--w---- 1 root tty 136, 6 Jan 29 09:22 6
c--------- 1 root root 5, 2 Jan 29 07:52 ptmx
The libvirt_lxc process that is spawned a link is created for /dev/console:
# ll /dev/conole
lrwxrwxrwx 1 root root 10 Jan 29 09:53 console -> /dev/pts/0
and /dev/pts/0 is also created:
# ll /dev/pts/0
crw--w---- 1 root tty 136, 0 Jan 29 10:05 /dev/pts/0
I'm surprised that the major/minor number for this isn't the same as
/dev/pts/2 in the host. I'm also surprised that no agetty process is
launched for the container. I'd expect to see something like this
running in the container:
# ps aux|grep agetty
root 25577 0.0 0.0 6424 792 pts/2 Ss+ 10:13 0:00
/sbin/agetty --noclear --keep-baud console 115200 38400 9600
I guess libvirt does some magic I'm not aware of to handle the consoles
for the containers. The question is why are we hitting this issue with
zombie processes that are caused by the console buffer filling up?
Peter