On 01/29/2016 05:08 AM, Peter Steele wrote: As I mentioned here, I think this may have to do with incorrect container configuration with regards to the console. Much of the process though is automated by libvirt itself so I'm not sure what I might be missing. When a container is created, the xml config has this entry defined: <console type='pty'> <target type='lxc' port='0'/> </console> After starting the container, the console config in the xml changes, e.g.: <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='lxc' port='0'/> <alias name='console0'/> </console> In addition to these changes, a new entry is created under /dev/pts: # ll /dev/pts crw--w---- 1 root tty 136, 0 Jan 29 08:27 0 crw--w---- 1 root tty 136, 1 Jan 29 08:26 1 crw--w---- 1 root tty 136, 2 Jan 29 09:19 2 <--- crw--w---- 1 root tty 136, 6 Jan 29 09:22 6 c--------- 1 root root 5, 2 Jan 29 07:52 ptmx The libvirt_lxc process that is spawned a link is created for /dev/console: # ll /dev/conole lrwxrwxrwx 1 root root 10 Jan 29 09:53 console -> /dev/pts/0 and /dev/pts/0 is also created: # ll /dev/pts/0 crw--w---- 1 root tty 136, 0 Jan 29 10:05 /dev/pts/0 I'm surprised that the major/minor number for this isn't the same as /dev/pts/2 in the host. I'm also surprised that no agetty process is launched for the container. I'd expect to see something like this running in the container: # ps aux|grep agetty root 25577 0.0 0.0 6424 792 pts/2 Ss+ 10:13 0:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 I guess libvirt does some magic I'm not aware of to handle the consoles for the containers. The question is why are we hitting this issue with zombie processes that are caused by the console buffer filling up? Peter

On 03/10/2016 12:34 AM, Martin Kletzander wrote: We're seeing this on CentOS 7.1, which is systemd based. We were able to determine that the cause of the problem is due to a container's console buffer being filled. In a container (or VM) the console of course is not a real physical device, it's a pseudo tty. With a physical console, when some process writes something to /dev/console, it appears on the physical console and if no one is there to see the text it eventually scrolls off the screen and is lost. There is no limit to how much text can be sent to the console. In the case of a container and its pseudo console, there is a buffer associated with the console device and this buffer has a size limit. If there is an active console session open for a container, any text sent to the container's console (e.g. by systemd) is consumed and processed by the container. However, if there is no active console session, as processes continue to write to the container's console device, the buffer associated with this pseudo console fills up. When this happens, any process that attempts to write to the container's console blocks and will stay blocked forever until a console session is started. These hung processes were the source of our zombie processes. We solved the problem by writing a console monitor service that runs on the hypervisor hosting the containers. It continually monitors the console devices of all containers and if there is an open console session for a given container, it does nothing. If however there is no active console session, it opens the console device for the container and drains it using the following Python code: fd = os.open(console, os.O_RDWR | os.O_NOCTTY) termios.tcflush(fd, termios.TCIFLUSH) os.close(fd); For expediency, we do not save the text that's read. This is ultimately similar to text scrolling off the top of a physical console. So, although this monitor service has solved our issue with zombie processes, I'm not convinced this is really the right solution. I'd like to think if a container is setup correctly, its console device should not fill up and block processes that attempt to write to it. I would think this would be a big problem for anyone running containers under libvirt_lxc. The problem is easy to reproduce in our environment: Open a console session to container and run "cat" with no arguments. Leave it running and disconnect the console session (control-]). Determine the container's console device from its xml definition, e.g. /dev/pts/3, and then copy some large file to it, e.g. # cp /var/log/messages /dev/pts/3 Assuming the file is larger than the console's backing buffer, this cp command should hang. If you then open a console session to this container from another window, you'll see the contents of /var/log/messages appear on the screen and the cp command in the other window will exit. If you are unable to reproduce it in your our setup following this procedure, then something is either wrong with my container configuration or there is something more insidious going on. I'd appreciate if you could run a test with this procedure and let me know the results. Thanks. Peter