12:12 a.m.
Greetings ...
I already spent quite some time to debug this ...
See the thread on gentoo-user ml for reference:
http://permalink.gmane.org/gmane.linux.gentoo.user/270091
basically I get dropped connections when I try to edit a VM via
virt-manager. Tested from my gentoo workstation and a fresh ubuntu
installation inside a VM at my office.
The server is a new and shiny gentoo box with (gentoo release-numbers):
libvirt-1.1.2-r3
qemu-1.5.3
openssh-5.9_p1-r4
...
64bit, lots of RAM and CPUs .. installed last week ... so no obvious
cruft on there.
plain ssh sessions work fine for me.
I access the server over an IPSEC-VPN terminated on my pfsense-router
and on the customers side there is a firewall. This firewall allows full
access for my private subnet, TCP/UDP ports 1:65535 ... so there should
be no restrictions.
Another issue: the server runs with systemd-204-r1 ... maybe important.
# systemctl status libvirtd
libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib64/systemd/system/libvirtd.service; enabled)
Active: active (running) since Do 2013-10-03 17:39:37 CEST; 13h ago
Main PID: 1242 (libvirtd)
CGroup: name=systemd:/system/libvirtd.service
└─1242 /usr/sbin/libvirtd --listen
Okt 03 17:39:38 jupiter libvirtd[1242]: [251B blob data]
Okt 03 17:39:38 jupiter libvirtd[1242]: [254B blob data]
Okt 03 17:39:38 jupiter libvirtd[1242]: [301B blob data]
Okt 03 17:39:38 jupiter libvirtd[1242]: firewall tools were not found or
cannot be used
Okt 03 17:39:41 jupiter libvirtd[1242]: [338B blob data]
Okt 03 17:39:41 jupiter libvirtd[1242]: failed to add iptables rule to
allow DHCP requests from 'virbr0'
Okt 03 17:41:07 jupiter libvirtd[1242]: No response from client
0x7fe8b0c19450 after 5 keepalive messages in 31 seconds
Okt 04 06:35:28 jupiter libvirtd[1242]: No response from client
0x7fe8b0c171a0 after 5 keepalive messages in 32 seconds
Okt 04 06:40:05 jupiter libvirtd[1242]: No response from client
0x7fe8b0c164e0 after 5 keepalive messages in 31 seconds
Okt 04 06:58:36 jupiter libvirtd[1242]: No response from client
0x7fe8b0c17050 after 5 keepalive messages in 31 seconds
What to install for the "firewall tools"? Gentoo didn't pull anything
related ... Is that relevant?
What about these keepalive messages, I assume this is responsible for my
failing virt-manager-connections.
Thanks for any help on this, I am quite stuck here already ...
Stefan
4:26 a.m.
Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
Thanks for any help on this, I am quite stuck here already ...
managed to edit the xml so far that I can boot from an iso ... still
have to edit stuff ...
In virt-viewer and virt-manager I don't have valid keyboard ... what
could be the reason?
I somehow wonder if the sockets used are somehow closed down or
something ... ?
S
9:03 a.m.
Am 04.10.2013 11:26, schrieb Stefan G. Weichinger:
Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
> Thanks for any help on this, I am quite stuck here already ...
managed to edit the xml so far that I can boot from an iso ... still
have to edit stuff ...
In virt-viewer and virt-manager I don't have valid keyboard ... what
could be the reason?
I somehow wonder if the sockets used are somehow closed down or
something ... ?
Additional thought: could it have to do with some IPv4/IPv6 topic?
The connection is IPv4 only ... but maybe the keepalive messages get
lost because the server tries to talk back via IPv6??
I dont know anything about that ... I will try that as soon as I have
access again. Right now I am on the road ...
Thanks for any help on this ...
2:36 p.m.
Am 04.10.2013 16:03, schrieb Stefan G. Weichinger:
Am 04.10.2013 11:26, schrieb Stefan G. Weichinger:
> Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
>
>> Thanks for any help on this, I am quite stuck here already ...
>
> managed to edit the xml so far that I can boot from an iso ... still
> have to edit stuff ...
>
> In virt-viewer and virt-manager I don't have valid keyboard ... what
> could be the reason?
>
> I somehow wonder if the sockets used are somehow closed down or
> something ... ?
Additional thought: could it have to do with some IPv4/IPv6 topic?
The connection is IPv4 only ... but maybe the keepalive messages get
lost because the server tries to talk back via IPv6??
I dont know anything about that ... I will try that as soon as I have
access again. Right now I am on the road ...
Thanks for any help on this ...
Anyone? Any hints? thanks ...
2:26 a.m.
On Mon, Oct 07, 2013 at 09:36:04PM +0200, Stefan G. Weichinger wrote:
Am 04.10.2013 16:03, schrieb Stefan G. Weichinger:
> Am 04.10.2013 11:26, schrieb Stefan G. Weichinger:
>> Am 04.10.2013 07:12, schrieb Stefan G. Weichinger:
>>
>>> Thanks for any help on this, I am quite stuck here already ...
>>
>> managed to edit the xml so far that I can boot from an iso ... still
>> have to edit stuff ...
>>
>> In virt-viewer and virt-manager I don't have valid keyboard ... what
>> could be the reason?
>>
For virt-manager and virt-viewer it is better to use virt-tools-list
[1], but the first issue you've reported isn't related to this, so I'm
not Cc'ing it there.
>> I somehow wonder if the sockets used are somehow closed down
or
>> something ... ?
>
Check the logs. You can set them according to a guide on logging [2]
that we have, because by default, I guess, you don't have debug logs
enabled. There will be *a lot* of noise, so try to filter unimportant
ones.
> Additional thought: could it have to do with some IPv4/IPv6
topic?
>
> The connection is IPv4 only ... but maybe the keepalive messages get
> lost because the server tries to talk back via IPv6??
>
I don't think so.
> I dont know anything about that ... I will try that as soon as I
have
> access again. Right now I am on the road ...
>
> Thanks for any help on this ...
Anyone? Any hints? thanks ...
And about the "firewall tools", you definitely want the
"virt-network"
use flag enabled. This is also not related to the problem you're
having, but will save you a lot of headache.
Try enabling the flag, re-emerging the package, setting the logs and
then reproduce it again. Check the logs and you should see why it's
disconnecting.
Martin
[1] https://www.redhat.com/mailman/listinfo/virt-tools-list
[2] http://libvirt.org/logging.html
7:46 a.m.
Am 08.10.2013 09:26, schrieb Martin Kletzander:
For virt-manager and virt-viewer it is better to use virt-tools-list
[1], but the first issue you've reported isn't related to this, so I'm
not Cc'ing it there.
OK, I consider posting there after trying the suggested steps.
Check the logs. You can set them according to a guide on logging
[2]
that we have, because by default, I guess, you don't have debug logs
enabled. There will be *a lot* of noise, so try to filter unimportant
ones.
Will do asap I am alone on the server again (people test stuff right now).
And about the "firewall tools", you definitely want the
"virt-network"
use flag enabled. This is also not related to the problem you're
having, but will save you a lot of headache.
Try enabling the flag, re-emerging the package, setting the logs and
then reproduce it again. Check the logs and you should see why it's
disconnecting.
I have that flag already, thanks.
More later this evening.
Thank you so far, Stefan
4:12 a.m.
Am 08.10.2013 14:46, schrieb Stefan G. Weichinger:
> Try enabling the flag, re-emerging the package, setting the logs
and
> then reproduce it again. Check the logs and you should see why it's
> disconnecting.
The docs say that libvirtd has to listen on the TCP port ... checked that:
# netstat -alnp | grep libv
tcp 0 0 0.0.0.0:16509 0.0.0.0:*
LISTEN 4568/libvirtd
libvirtd runs with UID root so I assume the user I use to connect with
has to be root as well?
I see no group-specification in libvirtd.conf, only for UNIX sockets ...
tested with my user sgw (in groups qemu, kvm, libvirt) and root ... same
behavior.
From my client I see the open port with nmap:
PORT STATE SERVICE
16509/tcp open unknown
so there should be no firewall topic ...
Increased logs on the libvirtd-server, so far it only says:
End of file while reading data: Input/output error
I have to play with the log settings now.
S
4:36 a.m.
Am 09.10.2013 11:12, schrieb Stefan G. Weichinger:
Increased logs on the libvirtd-server, so far it only says:
End of file while reading data: Input/output error
I have to play with the log settings now.
The firewall does not allow PING between server and client. Is that
necessary for the keepalive stuff?
S
4:49 a.m.
Here logs for a session wher I try to increase the RAM of one VM:
client: virt-manager --debug --no-fork
2013-10-09 11:44:08,903 (connection:579): Connection managed save
support: True
2013-10-09 11:44:11,212 (connection:161): Using libvirt API for netdev
enumeration
2013-10-09 11:44:11,213 (connection:201): Using libvirt API for mediadev
enumeration
/usr/share/virt-manager/virtManager/baseclass.py:240: GtkWarning:
Unknown property: GtkTable.halign
util.sanitize_gtkbuilder(self.uifile))
2013-10-09 11:44:15,363 (details:562): Showing VM details: <vmmDomain
object at 0x375e550 (virtManager+domain+vmmDomain at 0x354ba00)>
2013-10-09 11:44:15,408 (engine:327): window counter incremented to 2
2013-10-09 11:44:30,262 (libvirtobject:135): Redefining 'test-grml' with
XML diff:
--- Original XML
+++ New XML
@@ -1,7 +1,7 @@
<domain type="kvm">
<name>test-grml</name>
<uuid>faae6fdd-7366-f649-6582-fb2636cacae6</uuid>
- <memory unit="KiB">2097152</memory>
+ <memory unit="KiB">2098176</memory>
<currentMemory unit="KiB">2097152</currentMemory>
<vcpu placement="static">4</vcpu>
<os>
2013-10-09 11:45:01,211 (connection:1252): Unable to list inactive
domains: Interner Fehler: Ereignis zur Beendigung / Fehler am Socket
erhalten
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/connection.py", line 1250,
in _update_vms
newInactiveNames = self.vmm.listDefinedDomains()
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3660, in
listDefinedDomains
if ret is None: raise libvirtError ('virConnectListDefinedDomains()
failed', conn=self)
libvirtError: Interner Fehler: Ereignis zur Beendigung / Fehler am
Socket erhalten
2013-10-09 11:45:01,214 (connection:1419): Tick for VM 'audi' failed
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/connection.py", line 1417,
in _tick
vm.tick(now)
File "/usr/share/virt-manager/virtManager/domain.py", line 1664, in tick
info = self._backend.info()
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2021, in info
if ret is None: raise libvirtError ('virDomainGetInfo() failed',
dom=self)
libvirtError: Interner Fehler: Client Socket ist geschlossen
2013-10-09 11:45:01,214 (error:80): dialog message: Fehler beim Ändern
der VM-Konfiguration: Ende der Datei beim Lesen von Daten:
Eingabe-/Ausgabefehler : Fehler beim Ändern der VM-Konfiguration: Ende
der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/details.py", line 2544, in
_change_config_helper
self.vm.redefine_cached()
File "/usr/share/virt-manager/virtManager/domain.py", line 335, in
redefine_cached
self._redefine_xml(xml)
File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 147,
in _redefine_xml
return self._redefine_helper(origxml, newxml)
File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 137,
in _redefine_helper
self._define(newxml)
File "/usr/share/virt-manager/virtManager/domain.py", line 879, in _define
self.conn.define_domain(newxml)
File "/usr/share/virt-manager/virtManager/connection.py", line 814, in
define_domain
return self.vmm.defineXML(xml)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2924, in
defineXML
if ret is None:raise libvirtError('virDomainDefineXML() failed',
conn=self)
libvirtError: Ende der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler
----
server: # journalctl _SYSTEMD_UNIT=libvirtd.service -f
-- Logs begin at Tue 2013-08-27 23:38:28 CEST. --
Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '.'
Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '..'
Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '..'
Oct 09 11:25:13 jupiter libvirtd[11099]: Skipping special dir '.'
Oct 09 11:38:27 jupiter libvirtd[11099]: End of file while reading data:
Input/output error
Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '.'
Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '..'
Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '..'
Oct 09 11:39:42 jupiter libvirtd[11099]: Skipping special dir '.'
Oct 09 11:40:36 jupiter libvirtd[11099]: No response from client
0x7fbe6e11c970 after 5 keepalive messages in 30 seconds
Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '.'
Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '..'
Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '..'
Oct 09 11:43:53 jupiter libvirtd[11099]: Skipping special dir '.'
Oct 09 11:44:56 jupiter libvirtd[11099]: No response from client
0x7fbe6e11b780 after 5 keepalive messages in 31 seconds
---
when I restart libvirtd:
# journalctl _SYSTEMD_UNIT=libvirtd.service -f
-- Logs begin at Tue 2013-08-27 23:38:28 CEST. --
Oct 09 11:46:10 jupiter libvirtd[11311]: Reloading iptables rules
Oct 09 11:46:10 jupiter libvirtd[11311]: Refreshing network daemons
Oct 09 11:46:10 jupiter libvirtd[11311]: firewalld support disabled for
nwfilter
Oct 09 11:46:10 jupiter libvirtd[11311]: [249B blob data]
Oct 09 11:46:10 jupiter libvirtd[11311]: [138B blob data]
Oct 09 11:46:10 jupiter libvirtd[11311]: [253B blob data]
Oct 09 11:46:10 jupiter libvirtd[11311]: [251B blob data]
Oct 09 11:46:10 jupiter libvirtd[11311]: [254B blob data]
Oct 09 11:46:10 jupiter libvirtd[11311]: [301B blob data]
Oct 09 11:46:10 jupiter libvirtd[11311]: firewall tools were not found
or cannot be used
Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for configs in
/var/run/libvirt/qemu
Oct 09 11:46:11 jupiter libvirtd[11311]: Loading config file 'audi.xml'
Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for configs in
/etc/libvirt/qemu
Oct 09 11:46:11 jupiter libvirtd[11311]: Loading config file 'test-grml.xml'
Oct 09 11:46:11 jupiter libvirtd[11311]: Loading config file 'audi.xml'
Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for snapshots for
domain audi in /var/lib/libvirt/qemu/snapshot/audi
Oct 09 11:46:11 jupiter libvirtd[11311]: Scanning for snapshots for
domain test-grml in /var/lib/libvirt/qemu/snapshot/test-grml
Oct 09 11:46:14 jupiter libvirtd[11311]: firewall-cmd found but disabled
for iptables
Oct 09 11:46:14 jupiter libvirtd[11311]: [338B blob data]
Oct 09 11:46:14 jupiter libvirtd[11311]: failed to add iptables rule to
allow DHCP requests from 'virbr0'
Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '.'
Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '..'
Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '..'
Oct 09 11:46:17 jupiter libvirtd[11311]: Skipping special dir '.'
*sigh*
5:31 a.m.
On Wed, Oct 09, 2013 at 11:12:47AM +0200, Stefan G. Weichinger wrote:
Am 08.10.2013 14:46, schrieb Stefan G. Weichinger:
>> Try enabling the flag, re-emerging the package, setting the logs and
>> then reproduce it again. Check the logs and you should see why it's
>> disconnecting.
The docs say that libvirtd has to listen on the TCP port ... checked that:
# netstat -alnp | grep libv
tcp 0 0 0.0.0.0:16509 0.0.0.0:*
LISTEN 4568/libvirtd
Be sure to use some other authentication (e.g. sasl) when using tcp
(without TLS). Even better, use tls with sasl, but that's irrelevant
to the issue now.
libvirtd runs with UID root so I assume the user I use to connect
with
has to be root as well?
It's normal connection, it can't know under which user the client is
running.
I see no group-specification in libvirtd.conf, only for UNIX sockets
...
There is no way how to set permissions per group/user of the client,
see previous point.
tested with my user sgw (in groups qemu, kvm, libvirt) and root ...
same
behavior.
From my client I see the open port with nmap:
PORT STATE SERVICE
16509/tcp open unknown
so there should be no firewall topic ...
Increased logs on the libvirtd-server, so far it only says:
End of file while reading data: Input/output error
I have to play with the log settings now.
So let me revise that:
- you can reproduce it only with virt-manager not anything else
- the disconnection happens when you do what exactly?
- you tried setting the log level for both server and client
right?
I suggest the following:
set log_levev = 1 and log_outputs="1:file:/tmp/libvirtd.log" in your
libvirtd.conf, restart libvirtd, start virt-manager like this:
LIBVIRT_DEBUG=1 virt-manager --debug &>vm.log
and reproduce the problem. If this happens to you a lot (as it doesn
IIUC), feel free to use all the data to create a bug on upstream
virt-manager and we can continue from that.
Have a nice day,
Martin
3943
days inactive
3948
days old
9 comments
2 participants
participants (2)
-
Martin Kletzander -
Stefan G. Weichinger