[libvirt] serial console runs out of buffer and console messages soft lock the CPU

I recently started configuring guests to use both a serial console and a graphics element. One of the nodes with this configuration kept locking up after running for some time. Trying to access the guest through virt-manager would leave virt-viewer hung at the connecting to screen. The guest would not respond to any network traffic including icmp. The user would destroy the domain and restart it to recover. Eventually the problem came to me and I found that immediately upon running 'virsh console domain' the guest would recover. The console contained: BUG: soft lockup - CPU#0 stuck for 94393s! [postgresql-8.3:4975] I discovered a bug in a puppet recipe caused postgres to reload regularly, printing a message to the console each time. Running puppet in a loop to continue producing these messages to the console caused another lockup that was recoverable by accessing the serial console in the same manner. I have since fixed that recipe, but I'm uncomfortable about the possibility of this happening again. Is there a way to configure libvirt to cache the serial console on the host so the buffer on the guest does not fill up when you aren't connected to the console, or should I disable the serial console? Bryan https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/301897

On Tue, Nov 25, 2008 at 04:32:00PM -0800, Bryan McLellan wrote:
I recently started configuring guests to use both a serial console and a graphics element. One of the nodes with this configuration kept locking up after running for some time. Trying to access the guest through virt-manager would leave virt-viewer hung at the connecting to screen. The guest would not respond to any network traffic including icmp. The user would destroy the domain and restart it to recover. Eventually the problem came to me and I found that immediately upon running 'virsh console domain' the guest would recover. The console contained:
BUG: soft lockup - CPU#0 stuck for 94393s! [postgresql-8.3:4975]
I discovered a bug in a puppet recipe caused postgres to reload regularly, printing a message to the console each time. Running puppet in a loop to continue producing these messages to the console caused another lockup that was recoverable by accessing the serial console in the same manner. I have since fixed that recipe, but I'm uncomfortable about the possibility of this happening again.
Is there a way to configure libvirt to cache the serial console on the host so the buffer on the guest does not fill up when you aren't connected to the console, or should I disable the serial console?
This is actually a bug in QEMU. It should never block the VM when the serial port buffer fills up. Unfortunately when the serial port was connected to a TTY, QEMU would block on the write :-( The recent KVM releases fix this, but QEMU goes years without a release so it still suffers from this. I'd recommend filing a bug against QEMU in your distro. There's nothing libvirt can do to fix this. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Is there a way to configure libvirt to cache the serial console on the host so the buffer on the guest does not fill up when you aren't connected to the console, or should I disable the serial console? As Daniel pointed out this is a bug in KVM/Qemu. Debian got bitten by
On Tue, Nov 25, 2008 at 04:32:00PM -0800, Bryan McLellan wrote: this kvm bug too: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=494831 The bugreport has the necessary patches. Ubuntu just has to resync the Debian package again once these changes are in. Cheers, -- Guido
participants (3)
-
Bryan McLellan
-
Daniel P. Berrange
-
Guido Günther