[libvirt-users] failing connections w/ virt-manager
Greetings ... I already spent quite some time to debug this ... See the thread on gentoo-user ml for reference: http://permalink.gmane.org/gmane.linux.gentoo.user/270091 basically I get dropped connections when I try to edit a VM via virt-manager. Tested from my gentoo workstation and a fresh ubuntu installation inside a VM at my office. The server is a new and shiny gentoo box with (gentoo release-numbers): libvirt-1.1.2-r3 qemu-1.5.3 openssh-5.9_p1-r4 ... 64bit, lots of RAM and CPUs .. installed last week ... so no obvious cruft on there. plain ssh sessions work fine for me. I access the server over an IPSEC-VPN terminated on my pfsense-router and on the customers side there is a firewall. This firewall allows full access for my private subnet, TCP/UDP ports 1:65535 ... so there should be no restrictions. Another issue: the server runs with systemd-204-r1 ... maybe important. # systemctl status libvirtd libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib64/systemd/system/libvirtd.service; enabled) Active: active (running) since Do 2013-10-03 17:39:37 CEST; 13h ago Main PID: 1242 (libvirtd) CGroup: name=systemd:/system/libvirtd.service └─1242 /usr/sbin/libvirtd --listen Okt 03 17:39:38 jupiter libvirtd[1242]: [251B blob data] Okt 03 17:39:38 jupiter libvirtd[1242]: [254B blob data] Okt 03 17:39:38 jupiter libvirtd[1242]: [301B blob data] Okt 03 17:39:38 jupiter libvirtd[1242]: firewall tools were not found or cannot be used Okt 03 17:39:41 jupiter libvirtd[1242]: [338B blob data] Okt 03 17:39:41 jupiter libvirtd[1242]: failed to add iptables rule to allow DHCP requests from 'virbr0' Okt 03 17:41:07 jupiter libvirtd[1242]: No response from client 0x7fe8b0c19450 after 5 keepalive messages in 31 seconds Okt 04 06:35:28 jupiter libvirtd[1242]: No response from client 0x7fe8b0c171a0 after 5 keepalive messages in 32 seconds Okt 04 06:40:05 jupiter libvirtd[1242]: No response from client 0x7fe8b0c164e0 after 5 keepalive messages in 31 seconds Okt 04 06:58:36 jupiter libvirtd[1242]: No response from client 0x7fe8b0c17050 after 5 keepalive messages in 31 seconds What to install for the "firewall tools"? Gentoo didn't pull anything related ... Is that relevant? What about these keepalive messages, I assume this is responsible for my failing virt-manager-connections. Thanks for any help on this, I am quite stuck here already ... Stefan
Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
Thanks for any help on this, I am quite stuck here already ...
managed to edit the xml so far that I can boot from an iso ... still have to edit stuff ... In virt-viewer and virt-manager I don't have valid keyboard ... what could be the reason? I somehow wonder if the sockets used are somehow closed down or something ... ? S
Am 04.10.2013 11:26, schrieb Stefan G. Weichinger:
Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
Thanks for any help on this, I am quite stuck here already ...
managed to edit the xml so far that I can boot from an iso ... still have to edit stuff ...
In virt-viewer and virt-manager I don't have valid keyboard ... what could be the reason?
I somehow wonder if the sockets used are somehow closed down or something ... ?
Additional thought: could it have to do with some IPv4/IPv6 topic? The connection is IPv4 only ... but maybe the keepalive messages get lost because the server tries to talk back via IPv6?? I dont know anything about that ... I will try that as soon as I have access again. Right now I am on the road ... Thanks for any help on this ...
Am 04.10.2013 16:03, schrieb Stefan G. Weichinger:
Am 04.10.2013 11:26, schrieb Stefan G. Weichinger:
Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
Thanks for any help on this, I am quite stuck here already ...
managed to edit the xml so far that I can boot from an iso ... still have to edit stuff ...
In virt-viewer and virt-manager I don't have valid keyboard ... what could be the reason?
I somehow wonder if the sockets used are somehow closed down or something ... ?
Additional thought: could it have to do with some IPv4/IPv6 topic?
The connection is IPv4 only ... but maybe the keepalive messages get lost because the server tries to talk back via IPv6??
I dont know anything about that ... I will try that as soon as I have access again. Right now I am on the road ...
Thanks for any help on this ...
Anyone? Any hints? thanks ...
On Mon, Oct 07, 2013 at 09:36:04PM +0200, Stefan G. Weichinger wrote:
Am 04.10.2013 16:03, schrieb Stefan G. Weichinger:
Am 04.10.2013 11:26, schrieb Stefan G. Weichinger:
Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
Thanks for any help on this, I am quite stuck here already ...
managed to edit the xml so far that I can boot from an iso ... still have to edit stuff ...
In virt-viewer and virt-manager I don't have valid keyboard ... what could be the reason?
For virt-manager and virt-viewer it is better to use virt-tools-list [1], but the first issue you've reported isn't related to this, so I'm not Cc'ing it there.
I somehow wonder if the sockets used are somehow closed down or something ... ?
Check the logs. You can set them according to a guide on logging [2] that we have, because by default, I guess, you don't have debug logs enabled. There will be *a lot* of noise, so try to filter unimportant ones.
Additional thought: could it have to do with some IPv4/IPv6 topic?
The connection is IPv4 only ... but maybe the keepalive messages get lost because the server tries to talk back via IPv6??
I don't think so.
I dont know anything about that ... I will try that as soon as I have access again. Right now I am on the road ...
Thanks for any help on this ...
Anyone? Any hints? thanks ...
And about the "firewall tools", you definitely want the "virt-network" use flag enabled. This is also not related to the problem you're having, but will save you a lot of headache. Try enabling the flag, re-emerging the package, setting the logs and then reproduce it again. Check the logs and you should see why it's disconnecting. Martin [1] https://www.redhat.com/mailman/listinfo/virt-tools-list [2] http://libvirt.org/logging.html
Am 08.10.2013 09:26, schrieb Martin Kletzander:
For virt-manager and virt-viewer it is better to use virt-tools-list [1], but the first issue you've reported isn't related to this, so I'm not Cc'ing it there.
OK, I consider posting there after trying the suggested steps.
Check the logs. You can set them according to a guide on logging [2] that we have, because by default, I guess, you don't have debug logs enabled. There will be *a lot* of noise, so try to filter unimportant ones.
Will do asap I am alone on the server again (people test stuff right now).
And about the "firewall tools", you definitely want the "virt-network" use flag enabled. This is also not related to the problem you're having, but will save you a lot of headache.
Try enabling the flag, re-emerging the package, setting the logs and then reproduce it again. Check the logs and you should see why it's disconnecting.
I have that flag already, thanks. More later this evening. Thank you so far, Stefan
Am 08.10.2013 14:46, schrieb Stefan G. Weichinger:
Try enabling the flag, re-emerging the package, setting the logs and then reproduce it again. Check the logs and you should see why it's disconnecting.
The docs say that libvirtd has to listen on the TCP port ... checked that: # netstat -alnp | grep libv tcp 0 0 0.0.0.0:16509 0.0.0.0:* LISTEN 4568/libvirtd libvirtd runs with UID root so I assume the user I use to connect with has to be root as well? I see no group-specification in libvirtd.conf, only for UNIX sockets ... tested with my user sgw (in groups qemu, kvm, libvirt) and root ... same behavior.
From my client I see the open port with nmap:
PORT STATE SERVICE 16509/tcp open unknown so there should be no firewall topic ... Increased logs on the libvirtd-server, so far it only says: End of file while reading data: Input/output error I have to play with the log settings now. S
Am 09.10.2013 11:12, schrieb Stefan G. Weichinger:
Increased logs on the libvirtd-server, so far it only says:
End of file while reading data: Input/output error
I have to play with the log settings now.
The firewall does not allow PING between server and client. Is that necessary for the keepalive stuff? S
Here logs for a session wher I try to increase the RAM of one VM: client: virt-manager --debug --no-fork 2013-10-09 11:44:08,903 (connection:579): Connection managed save support: True 2013-10-09 11:44:11,212 (connection:161): Using libvirt API for netdev enumeration 2013-10-09 11:44:11,213 (connection:201): Using libvirt API for mediadev enumeration /usr/share/virt-manager/virtManager/baseclass.py:240: GtkWarning: Unknown property: GtkTable.halign util.sanitize_gtkbuilder(self.uifile)) 2013-10-09 11:44:15,363 (details:562): Showing VM details: <vmmDomain object at 0x375e550 (virtManager+domain+vmmDomain at 0x354ba00)> 2013-10-09 11:44:15,408 (engine:327): window counter incremented to 2 2013-10-09 11:44:30,262 (libvirtobject:135): Redefining 'test-grml' with XML diff: --- Original XML +++ New XML @@ -1,7 +1,7 @@ <domain type="kvm"> <name>test-grml</name> <uuid>faae6fdd-7366-f649-6582-fb2636cacae6</uuid> - <memory unit="KiB">2097152</memory> + <memory unit="KiB">2098176</memory> <currentMemory unit="KiB">2097152</currentMemory> <vcpu placement="static">4</vcpu> <os> 2013-10-09 11:45:01,211 (connection:1252): Unable to list inactive domains: Interner Fehler: Ereignis zur Beendigung / Fehler am Socket erhalten Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/connection.py", line 1250, in _update_vms newInactiveNames = self.vmm.listDefinedDomains() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3660, in listDefinedDomains if ret is None: raise libvirtError ('virConnectListDefinedDomains() failed', conn=self) libvirtError: Interner Fehler: Ereignis zur Beendigung / Fehler am Socket erhalten 2013-10-09 11:45:01,214 (connection:1419): Tick for VM 'audi' failed Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/connection.py", line 1417, in _tick vm.tick(now) File "/usr/share/virt-manager/virtManager/domain.py", line 1664, in tick info = self._backend.info() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2021, in info if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self) libvirtError: Interner Fehler: Client Socket ist geschlossen 2013-10-09 11:45:01,214 (error:80): dialog message: Fehler beim Ändern der VM-Konfiguration: Ende der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler : Fehler beim Ändern der VM-Konfiguration: Ende der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/details.py", line 2544, in _change_config_helper self.vm.redefine_cached() File "/usr/share/virt-manager/virtManager/domain.py", line 335, in redefine_cached self._redefine_xml(xml) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 147, in _redefine_xml return self._redefine_helper(origxml, newxml) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 137, in _redefine_helper self._define(newxml) File "/usr/share/virt-manager/virtManager/domain.py", line 879, in _define self.conn.define_domain(newxml) File "/usr/share/virt-manager/virtManager/connection.py", line 814, in define_domain return self.vmm.defineXML(xml) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2924, in defineXML if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self) libvirtError: Ende der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler ---- server: # journalctl _SYSTEMD_UNIT=libvirtd.service -f -- Logs begin at Tue 2013-08-27 23:38:28 CEST. -- Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '.' Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '..' Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '..' Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '.' Oct 09 11:38:27 jupiter libvirtd[11099]: End of file while reading data: Input/output error Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '.' Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '..' Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '..' Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '.' Oct 09 11:40:36 jupiter libvirtd[11099]: No response from client 0x7fbe6e11c970 after 5 keepalive messages in 30 seconds Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '.' Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '..' Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '..' Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '.' Oct 09 11:44:56 jupiter libvirtd[11099]: No response from client 0x7fbe6e11b780 after 5 keepalive messages in 31 seconds --- when I restart libvirtd: # journalctl _SYSTEMD_UNIT=libvirtd.service -f -- Logs begin at Tue 2013-08-27 23:38:28 CEST. -- Oct 09 11:46:10 jupiter libvirtd[11311]: Reloading iptables rules Oct 09 11:46:10 jupiter libvirtd[11311]: Refreshing network daemons Oct 09 11:46:10 jupiter libvirtd[11311]: firewalld support disabled for nwfilter Oct 09 11:46:10 jupiter libvirtd[11311]: [249B blob data] Oct 09 11:46:10 jupiter libvirtd[11311]: [138B blob data] Oct 09 11:46:10 jupiter libvirtd[11311]: [253B blob data] Oct 09 11:46:10 jupiter libvirtd[11311]: [251B blob data] Oct 09 11:46:10 jupiter libvirtd[11311]: [254B blob data] Oct 09 11:46:10 jupiter libvirtd[11311]: [301B blob data] Oct 09 11:46:10 jupiter libvirtd[11311]: firewall tools were not found or cannot be used Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for configs in /var/run/libvirt/qemu Oct 09 11:46:11 jupiter libvirtd[11311]: Loading config file 'audi.xml' Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for configs in /etc/libvirt/qemu Oct 09 11:46:11 jupiter libvirtd[11311]: Loading config file 'test-grml.xml' Oct 09 11:46:11 jupiter libvirtd[11311]: Loading config file 'audi.xml' Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for snapshots for domain audi in /var/lib/libvirt/qemu/snapshot/audi Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for snapshots for domain test-grml in /var/lib/libvirt/qemu/snapshot/test-grml Oct 09 11:46:14 jupiter libvirtd[11311]: firewall-cmd found but disabled for iptables Oct 09 11:46:14 jupiter libvirtd[11311]: [338B blob data] Oct 09 11:46:14 jupiter libvirtd[11311]: failed to add iptables rule to allow DHCP requests from 'virbr0' Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '.' Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '..' Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '..' Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '.' *sigh*
On Wed, Oct 09, 2013 at 11:12:47AM +0200, Stefan G. Weichinger wrote:
Am 08.10.2013 14:46, schrieb Stefan G. Weichinger:
Try enabling the flag, re-emerging the package, setting the logs and then reproduce it again. Check the logs and you should see why it's disconnecting.
The docs say that libvirtd has to listen on the TCP port ... checked that:
# netstat -alnp | grep libv tcp 0 0 0.0.0.0:16509 0.0.0.0:* LISTEN 4568/libvirtd
Be sure to use some other authentication (e.g. sasl) when using tcp (without TLS). Even better, use tls with sasl, but that's irrelevant to the issue now.
libvirtd runs with UID root so I assume the user I use to connect with has to be root as well?
It's normal connection, it can't know under which user the client is running.
I see no group-specification in libvirtd.conf, only for UNIX sockets ...
There is no way how to set permissions per group/user of the client, see previous point.
tested with my user sgw (in groups qemu, kvm, libvirt) and root ... same behavior.
From my client I see the open port with nmap:
PORT STATE SERVICE 16509/tcp open unknown
so there should be no firewall topic ...
Increased logs on the libvirtd-server, so far it only says:
End of file while reading data: Input/output error
I have to play with the log settings now.
So let me revise that: - you can reproduce it only with virt-manager not anything else - the disconnection happens when you do what exactly? - you tried setting the log level for both server and client right? I suggest the following: set log_levev = 1 and log_outputs="1:file:/tmp/libvirtd.log" in your libvirtd.conf, restart libvirtd, start virt-manager like this: LIBVIRT_DEBUG=1 virt-manager --debug &>vm.log and reproduce the problem. If this happens to you a lot (as it doesn IIUC), feel free to use all the data to create a bug on upstream virt-manager and we can continue from that. Have a nice day, Martin
participants (2)
-
Martin Kletzander -
Stefan G. Weichinger