[libvirt] [Question] capabilities.pidfile is left behind while starting and stopping libvirtd repeatly
Hi, all I started and stopped libvirtd service repeatly with high frequency(1 per second), and found that the file capabilities.pidfile is left behind, as well as a qemu process. If I then restart libvirtd, qemu-kvm will fail to start as it's unable to flock capabilities.pidfile's fd. Steps to reproduce the problem: 1. start libvirtd service per second with a shell script. 2. meanwhile, stop libvirtd service per second with another shell script. 3. then, a process qemu is left behind: /usr/bin/qemu-kvm -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 4. file /var/lib/libvirt/qemu/capabilities.pidfile is left behind 5. start libvirtd again, the process qemu-kvm fails to start. The cause of the problem: This file is generated by qemu, and deleted by libvirtd. If libvirtd got killed before it removes the pidfile, it would be left behind then. Question: Would it be fine if I kill qemu-kvm process and unlink capabilities.pidfile, just before virQEMUCapsInitQMP runs this qemu-kvm process? -- Best Regards Wang Yufei
On Tue, Sep 30, 2014 at 05:34:54PM +0800, Wang Yufei wrote:
Hi, all
I started and stopped libvirtd service repeatly with high frequency(1 per second), and found that the file capabilities.pidfile is left behind, as well as a qemu process. If I then restart libvirtd, qemu-kvm will fail to start as it's unable to flock capabilities.pidfile's fd.
Have you tried current master? Or at least -rc2? It should already be fixed there: commit 9e159b521dbf18c6da6976e54e29c8553f831eb6 Author: Guido Günther <agx@sigxcpu.org> Date: Thu Sep 25 10:30:58 2014 +0200 qemu: remove capabilities.monitor.sock when done Martin
Steps to reproduce the problem: 1. start libvirtd service per second with a shell script. 2. meanwhile, stop libvirtd service per second with another shell script. 3. then, a process qemu is left behind: /usr/bin/qemu-kvm -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 4. file /var/lib/libvirt/qemu/capabilities.pidfile is left behind 5. start libvirtd again, the process qemu-kvm fails to start.
The cause of the problem: This file is generated by qemu, and deleted by libvirtd. If libvirtd got killed before it removes the pidfile, it would be left behind then.
Question: Would it be fine if I kill qemu-kvm process and unlink capabilities.pidfile, just before virQEMUCapsInitQMP runs this qemu-kvm process?
-- Best Regards
Wang Yufei
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Sep 30, 2014 at 04:47:14PM +0200, Martin Kletzander wrote:
On Tue, Sep 30, 2014 at 05:34:54PM +0800, Wang Yufei wrote:
Hi, all
I started and stopped libvirtd service repeatly with high frequency(1 per second), and found that the file capabilities.pidfile is left behind, as well as a qemu process. If I then restart libvirtd, qemu-kvm will fail to start as it's unable to flock capabilities.pidfile's fd.
Have you tried current master? Or at least -rc2? It should already be fixed there:
commit 9e159b521dbf18c6da6976e54e29c8553f831eb6 Author: Guido Günther <agx@sigxcpu.org> Date: Thu Sep 25 10:30:58 2014 +0200
qemu: remove capabilities.monitor.sock when done
This one removes the monitor socket, not the pidfile since I didn't see that one lingering around in my tests. Cheers, -- Guido
On Tue, Sep 30, 2014 at 10:26:21PM +0200, Guido Günther wrote:
On Tue, Sep 30, 2014 at 04:47:14PM +0200, Martin Kletzander wrote:
On Tue, Sep 30, 2014 at 05:34:54PM +0800, Wang Yufei wrote:
Hi, all
I started and stopped libvirtd service repeatly with high frequency(1 per second), and found that the file capabilities.pidfile is left behind, as well as a qemu process. If I then restart libvirtd, qemu-kvm will fail to start as it's unable to flock capabilities.pidfile's fd.
Have you tried current master? Or at least -rc2? It should already be fixed there:
commit 9e159b521dbf18c6da6976e54e29c8553f831eb6 Author: Guido Günther <agx@sigxcpu.org> Date: Thu Sep 25 10:30:58 2014 +0200
qemu: remove capabilities.monitor.sock when done
This one removes the monitor socket, not the pidfile since I didn't see that one lingering around in my tests. Cheers,
Oh, sorry for that, my fault. Well, should the daemon just kill the process when it's dying in order to release the lock? If you try slower, we will properly remove the pidfile, won't we? Martin
On 2014/10/2 17:55, Martin Kletzander wrote:
On Tue, Sep 30, 2014 at 10:26:21PM +0200, Guido Günther wrote:
On Tue, Sep 30, 2014 at 04:47:14PM +0200, Martin Kletzander wrote:
On Tue, Sep 30, 2014 at 05:34:54PM +0800, Wang Yufei wrote:
Hi, all
I started and stopped libvirtd service repeatly with high frequency(1 per second), and found that the file capabilities.pidfile is left behind, as well as a qemu process. If I then restart libvirtd, qemu-kvm will fail to start as it's unable to flock capabilities.pidfile's fd.
Have you tried current master? Or at least -rc2? It should already be fixed there:
commit 9e159b521dbf18c6da6976e54e29c8553f831eb6 Author: Guido Günther <agx@sigxcpu.org> Date: Thu Sep 25 10:30:58 2014 +0200
qemu: remove capabilities.monitor.sock when done
This one removes the monitor socket, not the pidfile since I didn't see that one lingering around in my tests. Cheers,
Oh, sorry for that, my fault. Well, should the daemon just kill the process when it's dying in order to release the lock? If you try slower, we will properly remove the pidfile, won't we?
Yes, the pidfile will be keep open and locked forever if this qemu process is alive. And it will be unlink if I shutdown libvitd daemon slower. But in this condition(started and stopped libvirtd service repeatly with high frequency) the pidfile is left behind. In fact libvirtd daemon is killed before qemu-kvm process and pidfile are cleaned up. Is it a problem?
On Wed, Oct 08, 2014 at 04:07:33PM +0800, Wang Rui wrote:
On 2014/10/2 17:55, Martin Kletzander wrote:
On Tue, Sep 30, 2014 at 10:26:21PM +0200, Guido Günther wrote:
On Tue, Sep 30, 2014 at 04:47:14PM +0200, Martin Kletzander wrote:
On Tue, Sep 30, 2014 at 05:34:54PM +0800, Wang Yufei wrote:
Hi, all
I started and stopped libvirtd service repeatly with high frequency(1 per second), and found that the file capabilities.pidfile is left behind, as well as a qemu process. If I then restart libvirtd, qemu-kvm will fail to start as it's unable to flock capabilities.pidfile's fd.
Have you tried current master? Or at least -rc2? It should already be fixed there:
commit 9e159b521dbf18c6da6976e54e29c8553f831eb6 Author: Guido Günther <agx@sigxcpu.org> Date: Thu Sep 25 10:30:58 2014 +0200
qemu: remove capabilities.monitor.sock when done
This one removes the monitor socket, not the pidfile since I didn't see that one lingering around in my tests. Cheers,
Oh, sorry for that, my fault. Well, should the daemon just kill the process when it's dying in order to release the lock? If you try slower, we will properly remove the pidfile, won't we?
Yes, the pidfile will be keep open and locked forever if this qemu process is alive. And it will be unlink if I shutdown libvitd daemon slower. But in this condition(started and stopped libvirtd service repeatly with high frequency) the pidfile is left behind. In fact libvirtd daemon is killed before qemu-kvm process and pidfile are cleaned up. Is it a problem?
Simple kill(getpid(), SIGTERM) was enough to find out this is exactly what's happening. I have a fix prepared for this, but when I tried checking whether it fixes everything, I realized I can't find any problem this causes. And that's strange. All domain capabilities were in place even though I cleared them from the cache. Is there any cause of this problem? Does something not work then? I just want to know to test it basically :) Thanks for finding that out, Martin
participants (4)
-
Guido Günther -
Martin Kletzander -
Wang Rui -
Wang Yufei