On Thu, Nov 13, 2014 at 01:55:06PM -0800, Surojit Pathak wrote: > Hi all, > > [Issue observed] > If we issue 'nova reboot <server>', we get to have the console output of the > latest bootup of the server only. The console output of the previous boot > for the same server vanishes due to truncation[1]. If we do reboot from > within the VM instance [ #sudo reboot ], or reboot the instance with 'virsh > reboot <instance>' the behavior is not the same, where the console.log keeps > increasing, with the new output being appended. > This loss of history makes some debugging scenario difficult due to lack of > information being available. > > Please point me to any solution/blueprint for this issue, if already > planned. Otherwise, please comment on my analysis and proposals as solution, > below - > > [Analysis] > Nova's libvirt driver on compute node tries to do a graceful restart of the > server instance, by attempting a soft_reboot first. If soft_reboot fails, it > attempts a hard_reboot. As part of soft_reboot, it brings down the instance > by calling shutdown(), and then calls createWithFlags() to bring this up. > Because of this, qemu-kvm process for the instance gets terminated and new > process is launched. In QEMU, the chardev file is opened with O_TRUNC, and > thus we lose the previous content of the console.log file. > On the other-hand, during 'virsh reboot <instance>', the same qemu-kvm > process continues, and libvirt actually does a qemuDomainSetFakeReboot(). > Thus the same file continues capturing the new console output as a > continuation into the same file. Nova and libvirt have support for issuing a graceful reboot via the QEMU guest agent. So if you make sure that is installed, and tell Nova to use it, then Nova won't have to stop & recreate the QEMU process and thus won't have the problem of overwriting the logs.