9:24 a.m.
Hi.
Since the update to libvirt-1.3.2, virsh keeps disconnecting from qemu:///session.
Weirdly enough, qemu:///session works fine for root, and qemu:///system works for both the
user and the root.
With the same configuration(tarball with files from /etc/libvirt attached), libvirt-1.3.1
works without problems.
From the virsh debug log(attached):
---
grep error virsh.log.1.3.2
2016-03-03 19:19:20.338+0000: 5589: error : virNetSocketReadWire:1613 : End of file while
reading data: Input/output error
2016-03-03 19:19:46.261+0000: 5578: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:22:59.308+0000: 14487: error : virNetSocketReadWire:1613 : End of file while
reading data: Input/output error
2016-03-03 19:23:29.247+0000: 14486: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:24:49.521+0000: 14487: error : virKeepAliveTimerInternal:143 : internal
error: connection closed due to keepalive timeout
2016-03-03 19:25:17.043+0000: 14486: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:25:47.110+0000: 14487: error : virNetSocketReadWire:1613 : End of file while
reading data: Input/output error
2016-03-03 19:26:18.573+0000: 14486: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:28:24.142+0000: 14665: error : virNetSocketReadWire:1613 : End of file while
reading data: Input/output error
2016-03-03 19:28:55.091+0000: 14664: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
---
From the virt-manager.log (attached):
---
[Thu, 03 Mar 2016 18:39:38 virt-manager 24388] DEBUG (connection:1310) Error polling
connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:07:14 virt-manager 5191] ERROR (create:667) Error setting create
wizard conn state.
if ret is None: raise libvirtError ('virConnectGetCapabilities() failed',
conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:07:15 virt-manager 5191] DEBUG (connection:1310) Error polling
connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:08:19 virt-manager 5191] DEBUG (connection:1310) Error polling
connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:11:19 virt-manager 5191] DEBUG (connection:1310) Error polling
connection qemu:///system
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
---
qemu:///system errors above are due to restarting libvirtd, once virt-manager connected to
it, it stayed connected.
This is with qemu-2.5.0.
I could try libvirt master from git, but will it break libvirt-python/virt-manager?
Btw, is building libvirt with '--enable-debug' sufficient for useful gdb debug
report, or do you need anything else?
Thank you.
Pedja
4:16 a.m.
On 04.03.2016 16:24, Predrag Ivanovic wrote:
Hi.
Since the update to libvirt-1.3.2, virsh keeps disconnecting from qemu:///session.
Weirdly enough, qemu:///session works fine for root, and qemu:///system works for both
the user and the root.
With the same configuration(tarball with files from /etc/libvirt attached), libvirt-1.3.1
works without problems.
From the virsh debug log(attached):
---
grep error virsh.log.1.3.2
2016-03-03 19:19:20.338+0000: 5589: error : virNetSocketReadWire:1613 : End of file while
reading data: Input/output error
2016-03-03 19:19:46.261+0000: 5578: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:22:59.308+0000: 14487: error : virNetSocketReadWire:1613 : End of file
while reading data: Input/output error
2016-03-03 19:23:29.247+0000: 14486: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:24:49.521+0000: 14487: error : virKeepAliveTimerInternal:143 : internal
error: connection closed due to keepalive timeout
2016-03-03 19:25:17.043+0000: 14486: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:25:47.110+0000: 14487: error : virNetSocketReadWire:1613 : End of file
while reading data: Input/output error
2016-03-03 19:26:18.573+0000: 14486: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
2016-03-03 19:28:24.142+0000: 14665: error : virNetSocketReadWire:1613 : End of file
while reading data: Input/output error
2016-03-03 19:28:55.091+0000: 14664: error : virConnectUnregisterCloseCallback:1282 :
Requested operation is not valid: A different callback was requested
This looks like a dying session daemon to me. You can check if my
suspicion is right by running the session daemon by hand. From a console
run:
$ /usr/sbin/libvirtd
Usually, connecting to qemu:///session does the same, except it appends
'--timeout=30' to tell the session daemon to die after 30 seconds of
inactivity. And here's the problem - there should be a keepalive running
preventing daemon from dying.
---
From the virt-manager.log (attached):
---
[Thu, 03 Mar 2016 18:39:38 virt-manager 24388] DEBUG (connection:1310) Error polling
connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:07:14 virt-manager 5191] ERROR (create:667) Error setting create
wizard conn state.
if ret is None: raise libvirtError ('virConnectGetCapabilities() failed',
conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:07:15 virt-manager 5191] DEBUG (connection:1310) Error polling
connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:08:19 virt-manager 5191] DEBUG (connection:1310) Error polling
connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:11:19 virt-manager 5191] DEBUG (connection:1310) Error polling
connection qemu:///system
if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
---
qemu:///system errors above are due to restarting libvirtd, once virt-manager connected
to it, it stayed connected.
This is with qemu-2.5.0.
I could try libvirt master from git, but will it break libvirt-python/virt-manager?
No. The APIs are backward compatible.
Btw, is building libvirt with '--enable-debug' sufficient for
useful gdb debug report, or do you need anything else?
The best would be to disable any optimization and include gdb debuging info:
libvirt.git $ export CFLAGS="-O0 -ggdb3" ; ./configure --enable-debug
&&
make
Michal
12:02 p.m.
On Sat, 05 Mar 2016 11:16:21 +0100
Michal Privoznik wrote:
>This looks like a dying session daemon to me. You can check if my
suspicion is right by running the session daemon by hand. From a console
run:
$ /usr/sbin/libvirtd
Usually, connecting to qemu:///session does the same, except it appends
'--timeout=30' to tell the session daemon to die after 30 seconds of
inactivity. And here's the problem - there should be a keepalive running
preventing daemon from dying.
libvirt-1.3.1
root@deus:~/bin#> ps aux | grep libvirtd
pedja 3122 0.0 0.0 1027672 2376 ? Sl Mar03 0:03 /usr/sbin/libvirtd
--timeout=30
root 19814 0.0 0.1 1298708 5500 ? Sl Mar03 0:00 /usr/sbin/libvirtd -d -v
-f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid
-------
libvirt-git-92ec2e5
root@deus:~/bin#> ps aux | grep libvirtd
root 13553 0.0 0.4 972948 16552 ? Sl 16:02 0:00 /usr/sbin/libvirtd -d -v
-f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid
pedja 13723 0.0 0.4 965248 18508 ? Sl 16:03 0:00 /usr/sbin/libvirtd
--timeout=30
-------
libvirt-1.3.2
root@deus:~/bin#> ps aux | grep libvirtd
root 16778 0.3 0.3 383212 15700 ? Sl 17:10 0:00 /usr/sbin/libvirtd -d -v
-f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid
pedja 16975 3.2 0.4 637588 17964 ? Sl 17:11 0:00 /usr/sbin/libvirtd
--timeout=30
Only the session daemon from libvirt-1.3.2 dies randomly, 1.3.1 and libvirt-git-92ec2e5
are stable (no disconnects).
I managed to hook gdb to session daemon above after a few tries, and gdb log ('thread
apply all bt')
and virsh.log are attached.
I'll be running libvirt from git master for now, but I have both it and 1.3.2 built
with debug symbols, if you need more information.
Pedja
3188
days inactive
3189
days old
2 comments
2 participants
participants (2)
-
Michal Privoznik -
Predrag Ivanovic