[libvirt-users] [libvirt-1.3.2]'Disconnected from qemu:///session due to I/O error'
Hi. Since the update to libvirt-1.3.2, virsh keeps disconnecting from qemu:///session. Weirdly enough, qemu:///session works fine for root, and qemu:///system works for both the user and the root. With the same configuration(tarball with files from /etc/libvirt attached), libvirt-1.3.1 works without problems. From the virsh debug log(attached): --- grep error virsh.log.1.3.2 2016-03-03 19:19:20.338+0000: 5589: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:19:46.261+0000: 5578: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:22:59.308+0000: 14487: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:23:29.247+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:24:49.521+0000: 14487: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout 2016-03-03 19:25:17.043+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:25:47.110+0000: 14487: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:26:18.573+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:28:24.142+0000: 14665: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:28:55.091+0000: 14664: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested --- From the virt-manager.log (attached): --- [Thu, 03 Mar 2016 18:39:38 virt-manager 24388] DEBUG (connection:1310) Error polling connection qemu:///session if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:07:14 virt-manager 5191] ERROR (create:667) Error setting create wizard conn state. if ret is None: raise libvirtError ('virConnectGetCapabilities() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:07:15 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///session if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:08:19 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///session if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:11:19 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///system if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed --- qemu:///system errors above are due to restarting libvirtd, once virt-manager connected to it, it stayed connected. This is with qemu-2.5.0. I could try libvirt master from git, but will it break libvirt-python/virt-manager? Btw, is building libvirt with '--enable-debug' sufficient for useful gdb debug report, or do you need anything else? Thank you. Pedja
On 04.03.2016 16:24, Predrag Ivanovic wrote:
Hi.
Since the update to libvirt-1.3.2, virsh keeps disconnecting from qemu:///session. Weirdly enough, qemu:///session works fine for root, and qemu:///system works for both the user and the root. With the same configuration(tarball with files from /etc/libvirt attached), libvirt-1.3.1 works without problems.
From the virsh debug log(attached): --- grep error virsh.log.1.3.2 2016-03-03 19:19:20.338+0000: 5589: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:19:46.261+0000: 5578: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:22:59.308+0000: 14487: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:23:29.247+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:24:49.521+0000: 14487: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout 2016-03-03 19:25:17.043+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:25:47.110+0000: 14487: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:26:18.573+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested 2016-03-03 19:28:24.142+0000: 14665: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error 2016-03-03 19:28:55.091+0000: 14664: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested
This looks like a dying session daemon to me. You can check if my suspicion is right by running the session daemon by hand. From a console run: $ /usr/sbin/libvirtd Usually, connecting to qemu:///session does the same, except it appends '--timeout=30' to tell the session daemon to die after 30 seconds of inactivity. And here's the problem - there should be a keepalive running preventing daemon from dying.
--- From the virt-manager.log (attached): --- [Thu, 03 Mar 2016 18:39:38 virt-manager 24388] DEBUG (connection:1310) Error polling connection qemu:///session if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:07:14 virt-manager 5191] ERROR (create:667) Error setting create wizard conn state. if ret is None: raise libvirtError ('virConnectGetCapabilities() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:07:15 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///session if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:08:19 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///session if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed [Thu, 03 Mar 2016 19:11:19 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///system if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) libvirtError: internal error: client socket is closed if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) libvirtError: internal error: client socket is closed --- qemu:///system errors above are due to restarting libvirtd, once virt-manager connected to it, it stayed connected.
This is with qemu-2.5.0.
I could try libvirt master from git, but will it break libvirt-python/virt-manager?
No. The APIs are backward compatible.
Btw, is building libvirt with '--enable-debug' sufficient for useful gdb debug report, or do you need anything else?
The best would be to disable any optimization and include gdb debuging info: libvirt.git $ export CFLAGS="-O0 -ggdb3" ; ./configure --enable-debug && make Michal
On Sat, 05 Mar 2016 11:16:21 +0100 Michal Privoznik wrote:
This looks like a dying session daemon to me. You can check if my suspicion is right by running the session daemon by hand. From a console run:
$ /usr/sbin/libvirtd
Usually, connecting to qemu:///session does the same, except it appends '--timeout=30' to tell the session daemon to die after 30 seconds of inactivity. And here's the problem - there should be a keepalive running preventing daemon from dying.
libvirt-1.3.1 root@deus:~/bin#> ps aux | grep libvirtd pedja 3122 0.0 0.0 1027672 2376 ? Sl Mar03 0:03 /usr/sbin/libvirtd --timeout=30 root 19814 0.0 0.1 1298708 5500 ? Sl Mar03 0:00 /usr/sbin/libvirtd -d -v -f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid ------- libvirt-git-92ec2e5 root@deus:~/bin#> ps aux | grep libvirtd root 13553 0.0 0.4 972948 16552 ? Sl 16:02 0:00 /usr/sbin/libvirtd -d -v -f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid pedja 13723 0.0 0.4 965248 18508 ? Sl 16:03 0:00 /usr/sbin/libvirtd --timeout=30 ------- libvirt-1.3.2 root@deus:~/bin#> ps aux | grep libvirtd root 16778 0.3 0.3 383212 15700 ? Sl 17:10 0:00 /usr/sbin/libvirtd -d -v -f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid pedja 16975 3.2 0.4 637588 17964 ? Sl 17:11 0:00 /usr/sbin/libvirtd --timeout=30 Only the session daemon from libvirt-1.3.2 dies randomly, 1.3.1 and libvirt-git-92ec2e5 are stable (no disconnects). I managed to hook gdb to session daemon above after a few tries, and gdb log ('thread apply all bt') and virsh.log are attached. I'll be running libvirt from git master for now, but I have both it and 1.3.2 built with debug symbols, if you need more information. Pedja
participants (2)
-
Michal Privoznik -
Predrag Ivanovic