Re: [libvirt] [PATCH v3 0/6] Introduce virCommandDoAsyncIO

On Mon, Feb 04, 2013 at 03:56:37PM +0100, Michal Privoznik wrote: There is something racy in this series of changes. About 50% of the time when running a domain save, I see warnings about closing invalid file descriptors & invalid watches from libvirtd 2013-02-06 11:41:43.766+0000: 19078: warning : virEventPollRemoveHandle:183 : Ignoring invalid remove watch -1 2013-02-06 11:41:43.766+0000: 19078: warning : virFileClose:65 : Tried to close invalid fd 25 The first log message has the following trace #3 0x00007ffff7499b91 in virEventPollRemoveHandle (watch=-1) at util/vireventpoll.c:183 #4 0x00007ffff748b3c0 in virCommandHandleReadWrite (watch=-1, fd=-1, events=events@entry=9, opaque=opaque@entry=0x7fffcc01e380) at util/vircommand.c:2236 #5 0x00007ffff748ef8b in virCommandWait (cmd=0x7fffcc01e380, exitstatus=exitstatus@entry=0x0) at util/vircommand.c:2501 #6 0x00007ffff749b367 in virFileWrapperFdClose (wfd=wfd@entry=0x7fffcc01e360) at util/virfile.c:288 #7 0x00007fffdf180eb7 in qemuDomainSaveMemory (driver=driver@entry=0x7fffd806dfb0, vm=vm@entry=0x7fffbc001510, path=path@entry=0x7fffcc01e4b0 "/home/berrange/src/virt/libvirt-tck/tck.img", domXML=<optimized out>, compressed=compressed@entry=0, was_running=was_running@entry=true, flags=flags@entry=0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_SAVE) at qemu/qemu_driver.c:2792 #8 0x00007fffdf1842b3 in qemuDomainSaveInternal (driver=driver@entry=0x7fffd806dfb0, vm=0x7fffbc001510, path=path@entry=0x7fffcc01e4b0 "/home/berrange/src/virt/libvirt-tck/tck.img", compressed=0, xmlin=xmlin@entry=0x0, flags=flags@entry=0, dom=0x7fffcc00a1a0) at qemu/qemu_driver.c:2901 #9 0x00007fffdf1851f2 in qemuDomainSaveFlags (dom=0x7fffcc00a1a0, path=0x7fffcc01e4b0 "/home/berrange/src/virt/libvirt-tck/tck.img", dxml=0x0, flags=0) at qemu/qemu_driver.c:3013 #10 0x00007ffff75426e9 in virDomainSave (domain=domain@entry=0x7fffcc00a1a0, to=0x7fffcc01e2a0 "/home/berrange/src/virt/libvirt-tck/tck.img") at libvirt.c:2617 #11 0x0000000000428bc2 in remoteDispatchDomainSave (server=<optimized out>, msg=<optimized out>, args=0x7fffcc01e2e0, rerr=0x7fffe87a4c50, client=<optimized out>) at remote_dispatch.h:4758 #12 remoteDispatchDomainSaveHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7fffe87a4c50, args=0x7fffcc01e2e0, ret=<optimized out>) at remote_dispatch.h:4736 #13 0x00007ffff75af942 in virNetServerProgramDispatchCall (msg=0x693e10, client=0x694f60, server=0x66b490, prog=0x690bd0) at rpc/virnetserverprogram.c:432 #14 virNetServerProgramDispatch (prog=0x690bd0, server=server@entry=0x66b490, client=0x694f60, msg=0x693e10) at rpc/virnetserverprogram.c:305 #15 0x00007ffff75a9b78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x66b490) at rpc/virnetserver.c:162 #16 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x66b490) at rpc/virnetserver.c:183 #17 0x00007ffff74c304e in virThreadPoolWorker (opaque=opaque@entry=0x65ac90) at util/virthreadpool.c:144 #18 0x00007ffff74c2706 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #19 0x00007ffff3e9dd15 in start_thread (arg=0x7fffe87a5700) at pthread_create.c:308 #20 0x00007ffff37c946d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 The second log messages has this trace #3 0x00007ffff749b178 in virFileClose (fdptr=fdptr@entry=0x7fffcc01e3f8, flags=flags@entry=VIR_FILE_CLOSE_PRESERVE_ERRNO) at util/virfile.c:65 #4 0x00007ffff748b3da in virCommandHandleReadWrite (watch=-1, fd=-1, events=events@entry=9, opaque=opaque@entry=0x7fffcc01e380) at util/vircommand.c:2239 #5 0x00007ffff748ef8b in virCommandWait (cmd=0x7fffcc01e380, exitstatus=exitstatus@entry=0x0) at util/vircommand.c:2501 #6 0x00007ffff749b367 in virFileWrapperFdClose (wfd=wfd@entry=0x7fffcc01e360) at util/virfile.c:288 #7 0x00007fffdf180eb7 in qemuDomainSaveMemory (driver=driver@entry=0x7fffd806dfb0, vm=vm@entry=0x7fffbc001510, path=path@entry=0x7fffcc01e4b0 "/home/berrange/src/virt/libvirt-tck/tck.img", domXML=<optimized out>, compressed=compressed@entry=0, was_running=was_running@entry=true, flags=flags@entry=0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_SAVE) at qemu/qemu_driver.c:2792 #8 0x00007fffdf1842b3 in qemuDomainSaveInternal (driver=driver@entry=0x7fffd806dfb0, vm=0x7fffbc001510, path=path@entry=0x7fffcc01e4b0 "/home/berrange/src/virt/libvirt-tck/tck.img", compressed=0, xmlin=xmlin@entry=0x0, flags=flags@entry=0, dom=0x7fffcc00a1a0) at qemu/qemu_driver.c:2901 #9 0x00007fffdf1851f2 in qemuDomainSaveFlags (dom=0x7fffcc00a1a0, path=0x7fffcc01e4b0 "/home/berrange/src/virt/libvirt-tck/tck.img", dxml=0x0, flags=0) at qemu/qemu_driver.c:3013 #10 0x00007ffff75426e9 in virDomainSave (domain=domain@entry=0x7fffcc00a1a0, to=0x7fffcc01e2a0 "/home/berrange/src/virt/libvirt-tck/tck.img") at libvirt.c:2617 #11 0x0000000000428bc2 in remoteDispatchDomainSave (server=<optimized out>, msg=<optimized out>, args=0x7fffcc01e2e0, rerr=0x7fffe87a4c50, client=<optimized out>) at remote_dispatch.h:4758 #12 remoteDispatchDomainSaveHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7fffe87a4c50, args=0x7fffcc01e2e0, ret=<optimized out>) at remote_dispatch.h:4736 #13 0x00007ffff75af942 in virNetServerProgramDispatchCall (msg=0x693e10, client=0x694f60, server=0x66b490, prog=0x690bd0) at rpc/virnetserverprogram.c:432 #14 virNetServerProgramDispatch (prog=0x690bd0, server=server@entry=0x66b490, client=0x694f60, msg=0x693e10) at rpc/virnetserverprogram.c:305 #15 0x00007ffff75a9b78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x66b490) at rpc/virnetserver.c:162 #16 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x66b490) at rpc/virnetserver.c:183 #17 0x00007ffff74c304e in virThreadPoolWorker (opaque=opaque@entry=0x65ac90) at util/virthreadpool.c:144 #18 0x00007ffff74c2706 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #19 0x00007ffff3e9dd15 in start_thread (arg=0x7fffe87a5700) at pthread_create.c:308 #20 0x00007ffff37c946d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|