[libvirt] libvirt RPC error

Hi, I'm using qemu+ssh://username@hostname/system as the remote URI. Libvirt seems to be communicating fine until some 2 minutes (we poll every 5 seconds) and then it throws up RPC error and many counters are wrong. But if I collect on localhost the counters seems to be coming fine. Test and testvm are guests on the local machine where as Win8 and Ubuntu* are remote URI's connected through ssh. What causes this error "libvir: RPC error : End of file while reading data: : Input/output error" and how do we fix it ? I've pasted the logfile inline. Localhost from which we are connecting: uname -aLinux iwf0044110 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux root@iwf0044110:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2/Linux2.6_64_debug# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS" root@iwf0044110:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2/Linux2.6_64_debug# virsh version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.0.0 Remote host: root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2# uname -a Linux iwf004461 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=11.10 DISTRIB_CODENAME=oneiric DISTRIB_DESCRIPTION="Ubuntu 11.10" root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2# virsh version Compiled against library: libvir 0.9.2 Using library: libvir 0.9.2 Using API: QEMU 0.9.2 Running hypervisor: QEMU 0.14.1 Note: libvirt.so not present, using /usr/lib/libvirt.so.0 libvir: error : no connection driver available for No connection for URI xen:/// No Xen on this system NOTE: 1 memory stats available out of 8 libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: QEMU error : Requested operation is not valid: cgroup CPU controller is not mounted NOTE: scheduler type not available libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:07 | testvm | Running 04:23:07 | test | Running 04:23:07 | Win8 | Running 04:23:07 | Ubuntu11.10 | Running 04:23:07 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:18 | testvm | Running 04:23:18 | test | Running 04:23:18 | Win8 | Running 04:23:18 | Ubuntu11.10 | Running 04:23:18 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:25 | testvm | Running 04:23:25 | test | Running 04:23:25 | Win8 | Running 04:23:25 | Ubuntu11.10 | Running 04:23:25 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:32 | testvm | Running 04:23:32 | test | Running 04:23:32 | Win8 | Running 04:23:32 | Ubuntu11.10 | Running 04:23:32 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:40 | testvm | Running 04:23:40 | test | Running 04:23:40 | Win8 | Running 04:23:40 | Ubuntu11.10 | Running 04:23:40 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:47 | testvm | Running 04:23:47 | test | Running 04:23:47 | Win8 | Running 04:23:47 | Ubuntu11.10 | Running 04:23:47 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:23:55 | testvm | Running 04:23:55 | test | Running 04:23:55 | Win8 | Running 04:23:55 | Ubuntu11.10 | Running 04:23:55 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:02 | testvm | Running 04:24:02 | test | Running 04:24:02 | Win8 | Running 04:24:02 | Ubuntu11.10 | Running 04:24:02 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:10 | testvm | Running 04:24:10 | test | Running 04:24:10 | Win8 | Running 04:24:10 | Ubuntu11.10 | Running 04:24:10 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:17 | testvm | Running 04:24:17 | test | Running 04:24:17 | Win8 | Running 04:24:17 | Ubuntu11.10 | Running 04:24:17 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:24 | testvm | Running 04:24:24 | test | Running 04:24:24 | Win8 | Running 04:24:24 | Ubuntu11.10 | Running 04:24:24 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:32 | testvm | Running 04:24:32 | test | Running 04:24:32 | Win8 | Running 04:24:32 | Ubuntu11.10 | Running 04:24:32 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:39 | testvm | Running 04:24:39 | test | Running 04:24:39 | Win8 | Running 04:24:39 | Ubuntu11.10 | Running 04:24:39 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:46 | testvm | Running 04:24:46 | test | Running 04:24:46 | Win8 | Running 04:24:46 | Ubuntu11.10 | Running 04:24:46 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:24:54 | testvm | Running 04:24:54 | test | Running 04:24:54 | Win8 | Running 04:24:54 | Ubuntu11.10 | Running 04:24:54 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:25:01 | testvm | Running 04:25:01 | test | Running 04:25:01 | Win8 | Running 04:25:01 | Ubuntu11.10 | Running 04:25:01 | Ubuntu12.04 | Running libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known libvir: error : internal error getaddrinfo failed for 'iwf004461': Name or service not known 04:25:09 | testvm | Running 04:25:09 | test | Running 04:25:09 | Win8 | Running 04:25:09 | Ubuntu11.10 | Running 04:25:09 | Ubuntu12.04 | Running libvir: RPC error : End of file while reading data: : Input/output error 04:25:16 | testvm | Running 04:25:16 | test | Running 04:25:16 | Win8 | Nostate 04:25:16 | Ubuntu11.10 | Nostate 04:25:16 | Ubuntu12.04 | Nostate libvir: RPC error : End of file while reading data: : Input/output error 04:25:20 | testvm | Running 04:25:20 | test | Running 04:25:20 | Win8 | Nostate 04:25:20 | Ubuntu11.10 | Nostate 04:25:20 | Ubuntu12.04 | Nostate libvir: RPC error : End of file while reading data: : Input/output error 04:25:25 | testvm | Running 04:25:25 | test | Running 04:25:25 | Win8 | Nostate 04:25:25 | Ubuntu11.10 | Nostate 04:25:25 | Ubuntu12.04 | Nostate Regards, Navin P.S. Please use reply all as I'm not subscribed to this list .

On 01/11/13 07:01, Parakkal, Navin S wrote:
Hi,
I’m using qemu+ssh://username@hostname/system as the remote URI. Libvirt seems to be communicating fine until some 2 minutes (we poll every 5 seconds) and then it throws up RPC error and many counters are wrong. But if I collect on localhost the counters seems to be coming fine. Test and testvm are guests on the local machine where as Win8 and Ubuntu* are remote URI’s connected through ssh.
What causes this error “libvir: RPC error : End of file while reading data: : Input/output error”
and how do we fix it ?
I’ve pasted the logfile inline.
Localhost from which we are connecting:
uname -aLinux iwf0044110 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@iwf0044110:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2/Linux2.6_64_debug# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"
root@iwf0044110:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2/Linux2.6_64_debug# virsh version
Compiled against library: libvir 0.9.8
Using library: libvir 0.9.8
Using API: QEMU 0.9.8
Running hypervisor: QEMU 1.0.0
Remote host:
root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2# uname -a
Linux iwf004461 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"
root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/linux/2.2# virsh version
Compiled against library: libvir 0.9.2
Using library: libvir 0.9.2
Using API: QEMU 0.9.2
Running hypervisor: QEMU 0.14.1
Note: libvirt.so not present, using /usr/lib/libvirt.so.0
libvir: error : no connection driver available for No connection for URI xen:///
No Xen on this system
NOTE: 1 memory stats available out of 8
Hm those errors you pasted seem really strange. As it appears to work on localhost, the problem will be probably in the transport itself. As the log you posted contains only error messages it's hard to tell the reason for this behavior. One of the reasons might be that the machine name isn't resolving correctly. Please try to fix your DNS or /etc/hosts file so that the resolution works. Could you please post logs containing debug logs both from the client (refer to http://libvirt.org/logging.html#log_library for steps to enable them) and also from the server. Please attach them compressed as they tend to grow really large after a short time. Thanks Peter

Hi, Does the transport have any configuration file to throttle the connections through ssh made from a client ? The logs are huge even after compressing(5MB). Is there a site where I can upload this to ? Regards, Navin -----Original Message----- From: Peter Krempa [mailto:pkrempa@redhat.com] Sent: Friday, January 11, 2013 2:23 PM To: Parakkal, Navin S Cc: libvir-list@redhat.com; libvirt-users@redhat.com; Pargaonkar, Vishwanath Bandoo (HP Software); Bertin, Chris R; Sudhakaran, Sonu (TSG Software) Subject: Re: [libvirt] libvirt RPC error On 01/11/13 07:01, Parakkal, Navin S wrote:
Hi,
I'm using qemu+ssh://username@hostname/system as the remote URI. Libvirt seems to be communicating fine until some 2 minutes (we poll every 5 seconds) and then it throws up RPC error and many counters are wrong. But if I collect on localhost the counters seems to be coming fine. Test and testvm are guests on the local machine where as Win8 and Ubuntu* are remote URI's connected through ssh.
What causes this error "libvir: RPC error : End of file while reading data: : Input/output error"
and how do we fix it ?
I've pasted the logfile inline.
Localhost from which we are connecting:
uname -aLinux iwf0044110 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
root@iwf0044110:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob /linux/2.2/Linux2.6_64_debug# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"
root@iwf0044110:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob /linux/2.2/Linux2.6_64_debug# virsh version
Compiled against library: libvir 0.9.8
Using library: libvir 0.9.8
Using API: QEMU 0.9.8
Running hypervisor: QEMU 1.0.0
Remote host:
root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/ linux/2.2# uname -a
Linux iwf004461 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/ linux/2.2# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"
root@iwf004461:/home1/parakkal/COLLABNET_11.10_NEW/hpsw-oa/PA/numsVob/ linux/2.2# virsh version
Compiled against library: libvir 0.9.2
Using library: libvir 0.9.2
Using API: QEMU 0.9.2
Running hypervisor: QEMU 0.14.1
Note: libvirt.so not present, using /usr/lib/libvirt.so.0
libvir: error : no connection driver available for No connection for URI xen:///
No Xen on this system
NOTE: 1 memory stats available out of 8
Hm those errors you pasted seem really strange. As it appears to work on localhost, the problem will be probably in the transport itself. As the log you posted contains only error messages it's hard to tell the reason for this behavior. One of the reasons might be that the machine name isn't resolving correctly. Please try to fix your DNS or /etc/hosts file so that the resolution works. Could you please post logs containing debug logs both from the client (refer to http://libvirt.org/logging.html#log_library for steps to enable them) and also from the server. Please attach them compressed as they tend to grow really large after a short time. Thanks Peter
participants (2)
-
Parakkal, Navin S
-
Peter Krempa