Re: [libvirt] [PATCH v2] Close the source fd if the destination qemu exits during tunnelled migration
On 09/29/2015 10:06 AM, Shivaprasad bhat wrote:
[...]
> Perhaps I should clarify my query-migrate has no timeout
comment... It
> seems based on what I've read so far, the 'query-migrate' command
> started successfully, because if it hadn't we would have received a
> failure (as shown below). Thus libvirt has sent the command via the
> monitor and is waiting for a response (e.g. the virCondWait after the
> qemuMonitorSend in the trace below). The response isn't coming because
> either "A" qemu didn't send it back or "B" libvirt missed it
- that
> should be determinable.
>
> There's a way to turn on debugging so the monitor dialog can be seen -
> via changes to /etc/libvirt/libvirtd.conf. I use :
>
> log_level = 1
> log_filters="3:remote 4:event 3:json 3:rpc"
> log_outputs="1:file:/var/log/libvirt/libvirtd.log"
>
> But you may need to remove the "3:json" in order to see the dialog since
> that where it "feels like" the issue might be. Then start libvirtd in
> the debugger again. Once it's hung - you should be able to scan (eg,
> edit) the libvirtd.log file and search for the "query-migrate" command
> being sent and then follow the copious output looking for the presence
> of a returned command. If there is none, then something in qemu isn't
> returning the failure correctly and it would need to be fixed there I
> would think as opposed to throwing down the big hammer of closing the fd.
Had a chance to run with your log settings. The query-migrate doesn't seem to
have a corresponding "return" in the logs. So as you say, there may be
a qemu bug
that is not returning a response when the fd is still open(as libvirt
didnt close it) but
no read actually happening there. I felt qemu can't sense the failure as the fd
is open, so posted this patch. Though, the qemu should return with the
current state
of migration as it sees instead of not returning at all. Hope we are
on the same page.
Thanks,
Shivaprasad
Meant to respond yesterday but got wrapped up in other things. OK - so
at least now it makes a bit more sense why purely adding a stream_abort
didn't work - we're not getting a reply from the monitor.
So there's perhaps 3 ways to "resolve" this issue (that come to my mind)
1. As you've done with the close() in the error path of
qemuMigrationIOFunc when the virStream{Send|Finish} fails. Although this
does feel like a work-around, I suppose since the tunnel is a libvirt
created thing and qemu isn't aware of it, then it feels reasonable.
Although that does make me wonder how qemu could be hung up. What would
something like a "virsh qemu-monitor-command $dom
'{"execute":"query-migrate"}' return when the source is hung?
Or does it
hang too?
2. Adding some sort of "timeout" logic in qemuMonitorSend (e.g.
virCondWaitUntil instead of virCondWait) to handle when a command
doesn't get a response. Not sure this is right either since it's not
clear to me there is a "time" that "all" commands are guaranteed to
be
run in/by, especially async ones.
3. Dig into qemu to figure out why it's not returning anything for a
migrate-status request. Currently a bit beyond what I've done, but I
believe would require attaching into the running qemu process to see if
there was some thread "stuck" somewhere "waiting" on something that
won't return because the stream closed.
Hopefully Jiri (or perhaps Daniel) could provide some other insights.
John