2:10 a.m.
On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote:
I have identical two hypervisors same operating system: Ubuntu 22.04.2
LTS
Recently both virsh stopped talking to the libvirtd. Both stopped within a
few days of each other.
Currently if I run:
virsh uri
virsh version
virsh list
# virsh list
..nothing just hangs
When I ran strace on these broken machines it get stuck at same spot:
Is libvirtd running? It might be that you have socket activation with
systemd and the socket this virsh is connecting to is not properly
associated with the service. One of the things that might happen is
that you want to debug the service, stop the service and a socket unit,
but that will not remove it. Before debugging this make sure everything
related in systemd is stopped and then try running libvirtd (or
virtqemud, there are two services) with debugging enabled and then run
the virsh commands with debugging enabled as well.
Martin
strace virsh list
...
access("/var/run/libvirt/virtqemud-sock", F_OK) = -1 ENOENT (No such file
or directory)
access("/var/run/libvirt/libvirt-sock", F_OK) = 0
socket(AF_UNIX, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"},
110) = 0
getsockname(5, {sa_family=AF_UNIX}, [128 => 2]) = 0
futex(0x7fa716a672f0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fcntl(5, F_GETFD) = 0
fcntl(5, F_SETFD, FD_CLOEXEC) = 0
fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
futex(0x7fa716a67348, FUTEX_WAKE_PRIVATE, 2147483647) = 0
eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 6
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
write(4, "\1\0\0\0\0\0\0\0", 8) = 8
write(4, "\1\0\0\0\0\0\0\0", 8) = 8
futex(0x7fa70c001cb0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fa716a6786c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fa716a67378, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(4, "\1\0\0\0\0\0\0\0", 8) = 8
futex(0x7fa70c001cb0, FUTEX_WAKE_PRIVATE, 1) = 1
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=5, events=POLLOUT}, {fd=6, events=POLLIN}], 2, -1) = 2 ([{fd=5,
revents=POLLOUT}, {fd=6, revents=POLLIN}])
read(6, "\2\0\0\0\0\0\0\0", 16) = 8
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
futex(0x5628ce6e9710, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(5, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0",
28) = 28
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1) = 1 ([{fd=6,
revents=POLLIN}])
read(6, "\5\0\0\0\0\0\0\0", 16) = 8
poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1
It gets stuck at this poll(). Note I tested strace on identical new
install of ubtuntu 22.04 where virsh connects fine and get identical
strace, except after this poll() it continues on with read/write ..etc.
I turned on debugging for libvirtd and get no errors while virsh is trying
to connect.
I am able to get a virsh# shell. The shell only hangs when I try "connect,
uri, version".
Another method of debugging I tried was:
LIBVIRT_DEBUG=error LIBVIRT_LOG_FILTERS="1:* " virsh uri
..
..
2023-06-06 20:51:22.312+0000: 1647: debug : doRemoteOpen:1128 : Trying
authentication
2023-06-06 20:51:22.312+0000: 1647: debug : virNetMessageNew:44 :
msg=0x55b996539680 tracked=0
2023-06-06 20:51:22.312+0000: 1647: debug : virNetMessageEncodePayload:383
: Encode length as 28
2023-06-06 20:51:22.312+0000: 1647: info : virNetClientSendInternal:2151 :
RPC_CLIENT_MSG_TX_QUEUE: client=0x55b996538010 len=28 prog=536903814
vers=1 proc=66 type=0 status=0 serial=0
2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientCallNew:2107 : New
call 0x55b996535f80: msg=0x55b996539680, expectReply=1, nonBlock=0
2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientIO:1920 : Outgoing
message prog=536903814 version=1 serial=0 proc=66 type=0 length=28
dispatch=(nil)
2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientIO:1978 : We have
the buck head=0x55b996535f80 call=0x55b996535f80
2023-06-06 20:51:22.312+0000: 1647: info : virEventGLibHandleUpdate:195 :
EVENT_GLIB_UPDATE_HANDLE: watch=1 events=0
2023-06-06 20:51:22.312+0000: 1647: debug : virEventGLibHandleUpdate:206 :
Update handle data=0x55b996534d30 watch=1 fd=5 events=0
2023-06-06 20:51:22.312+0000: 1647: debug : virEventGLibHandleUpdate:229 :
Removed old handle source=0x55b996534de0
2023-06-06 20:51:22.312+0000: 1648: debug : virEventRunDefaultImpl:341 :
running default event implementation
Any help would be appreciated.
thanks
jerry