2:57 a.m.
To rescue this bug from the noise in a subthread:
If a hostname resolves to more than one address and the host currently
configured itself for just IPv4, doing a bind() to some IPv6 address
will fail. As a result an error is returned instead of continuing with
the next item in 'runp'.
Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20
08:35:52.521+0000: 4531: error : virNetSocketNewListenTCP:389 : Unable to bind to port:
Cannot assign requested address
After further debugging:
27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35
27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152),
inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr),
sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested
address)
27672 16:04:46.775455 gettid() = 27672
27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+0000: 27672: error :
virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested
address\n", 132) = 132
27672 16:04:46.775742 gettid() = 27672
27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+0000: 27672: info :
virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98
27672 16:04:46.776026 gettid() = 27672
So for some reason libvirtd tries to bind to ipv6 even if the host does
not have that ipv6 address at this point, only the link-local address.
Not sure if there is a way to detect that within libvirt. Perhaps it
should just move on with the runp list and try the next one?
Olaf
3:56 a.m.
On Tue, Mar 27, 2018 at 09:57:13AM +0200, Olaf Hering wrote:
To rescue this bug from the noise in a subthread:
If a hostname resolves to more than one address and the host currently
configured itself for just IPv4, doing a bind() to some IPv6 address
will fail. As a result an error is returned instead of continuing with
the next item in 'runp'.
> Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+0000: 4531:
error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested
address
After further debugging:
27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35
27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152),
inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr),
sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested
address)
27672 16:04:46.775455 gettid() = 27672
27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+0000: 27672: error :
virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested
address\n", 132) = 132
27672 16:04:46.775742 gettid() = 27672
27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+0000: 27672: info :
virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98
27672 16:04:46.776026 gettid() = 27672
So for some reason libvirtd tries to bind to ipv6 even if the host does
not have that ipv6 address at this point, only the link-local address.
Not sure if there is a way to detect that within libvirt. Perhaps it
should just move on with the runp list and try the next one?
It looks like the virNetSocketNewListenTCP only treats EAFNOSUPPORT as an
error to continue with - everything else is fatal. At very least we need
to add EADDRNOTAVAIL too.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
4:10 a.m.
On Tue, Mar 27, 2018 at 09:57:13AM +0200, Olaf Hering wrote:
To rescue this bug from the noise in a subthread:
If a hostname resolves to more than one address and the host currently
configured itself for just IPv4, doing a bind() to some IPv6 address
will fail. As a result an error is returned instead of continuing with
the next item in 'runp'.
> Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+0000: 4531:
error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested
address
After further debugging:
27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35
27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152),
inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr),
sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested
address)
27672 16:04:46.775455 gettid() = 27672
27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+0000: 27672: error :
virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested
address\n", 132) = 132
27672 16:04:46.775742 gettid() = 27672
27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+0000: 27672: info :
virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98
27672 16:04:46.776026 gettid() = 27672
So for some reason libvirtd tries to bind to ipv6 even if the host does
not have that ipv6 address at this point, only the link-local address.
Not sure if there is a way to detect that within libvirt. Perhaps it
should just move on with the runp list and try the next one?
The AI_ADDRCONFIG flag to getaddrinfo should have taken care of
filtering out IPv6 addresses if you only have the link-local one
(fe80::), but 2620:: looks like a legitimate unicast address.
Checking for EADDRNOTAVAIL here is intentional to catch configuration
errors (i.e. a typo in a literal listen address or misconfigured DNS).
Why does your hostname resolve to an unavailable address?
Jano
4:18 a.m.
On Tue, Mar 27, Ján Tomko wrote:
Why does your hostname resolve to an unavailable address?
How can the DNS server possibly know how a host has configured itself?
In this case I had BOOTPROTO='dhcp4' instead of 'dhcp' in
/etc/sysconfig/network/ifcfg-br0 due to all the migration issues I'm
seeing. It turned out they are unrelated to such setting.
Olaf
12:46 p.m.
On Tue, Mar 27, 2018 at 11:18:10AM +0200, Olaf Hering wrote:
On Tue, Mar 27, Ján Tomko wrote:
> Why does your hostname resolve to an unavailable address?
How can the DNS server possibly know how a host has configured itself?
It cannot, but the admin of the network should be able to control both.
How can libvirt tell whether this is a misconfiguration of DNS or host's
interfaces?
But we can possibly apply different rules for different callers:
virNetServerServiceNewTCP where silently ignoring a failure for one
address on daemon/system startup might be hard to catch and
libxlDomainMigrationDstPrepare which was called by an API
or by data source - whether it was user provided in the migration URI
or libvirtd tried to figure it out
For QEMU migration, we allow overriding the default listen address
via the migrate_host qemu.conf option, this is passed to QEMU
and looking at the code, it goes for best-effort and does not report
an error as long as binding to one of the addresses succeeds.
Anyway, globally ignoring EADDRNOTAVAIL feels too lenient for me.
Jano
In this case I had BOOTPROTO='dhcp4' instead of 'dhcp'
in
/etc/sysconfig/network/ifcfg-br0 due to all the migration issues I'm
seeing. It turned out they are unrelated to such setting.
Olaf
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
8:56 a.m.
On Tue, Mar 27, Ján Tomko wrote:
It cannot, but the admin of the network should be able to control
both.
The admin must not control my (test) host, nor must I control the DNS
server in the network. But there are likely cases where the admin for
DNS and libvirtd is the same person.
How can libvirt tell whether this is a misconfiguration of DNS or
host's
interfaces?
By simply cycling through the 'runp' list to see if any bind() succeeds?
Olaf
3:37 p.m.
On Wed, Mar 28, Olaf Hering wrote:
> How can libvirt tell whether this is a misconfiguration of DNS
or host's
> interfaces?
By simply cycling through the 'runp' list to see if any bind() succeeds?
This change fixes /etc/sysconfig/network/ifcfg-br0:BOOTPROTO='dhcp4' for me.
Just keep going in virNetSocketNewListenTCP:
Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.553+0000: 8021: error :
virNetSocketNewListenTCP:323 : virNetSocketNewListenTCP: macintyre-old.arch.suse.de 49152
0: Success
Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.555+0000: 8021: error :
virNetSocketNewListenTCP:394 : virNetSocketNewListenTCP: bind to '10.161.8.197':
Ok: Success
Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.555+0000: 8021: error :
virNetSocketNewListenTCP:394 : virNetSocketNewListenTCP: bind to
'2620:113:80c0:8000:10:161:8:197': Ok: Cannot assign requested address
You get the idea.
Olaf
---
src/rpc/virnetsocket.c | 34 ++++++++++++++++++++--------------
1 file changed, 20 insertions(+), 14 deletions(-)
--- a/src/rpc/virnetsocket.c
+++ b/src/rpc/virnetsocket.c
@@ -320,6 +320,7 @@ int virNetSocketNewListenTCP(const char *nodename,
*retsocks = NULL;
*nretsocks = 0;
+ virReportSystemError(errno, "%s: %s %s %d", __func__, nodename, service,
family);
memset(&hints, 0, sizeof(hints));
hints.ai_family = family;
hints.ai_flags = AI_PASSIVE;
@@ -383,12 +384,17 @@ int virNetSocketNewListenTCP(const char *nodename,
}
}
#endif
-
- if (bind(fd, runp->ai_addr, runp->ai_addrlen) < 0) {
- if (errno != EADDRINUSE) {
- virReportSystemError(errno, "%s", _("Unable to bind to
port"));
- goto error;
- }
+ e = bind(fd, runp->ai_addr, runp->ai_addrlen);
+ {
+ char hostname[123];
+ int r, oe = errno;
+ memset(hostname, 0, sizeof(hostname));
+ r = getnameinfo(runp->ai_addr,runp->ai_addrlen,hostname, sizeof(hostname),
NULL, 0, NI_NUMERICHOST);
+ errno = oe;
+ virReportSystemError(errno, "%s: bind to '%s': %s", __func__,
hostname, r ? gai_strerror(r) : "Ok");
+ }
+ if (e < 0) {
+ if (errno == EADDRINUSE)
addrInUse = true;
VIR_FORCE_CLOSE(fd);
runp = runp->ai_next;
@@ -412,14 +418,14 @@ int virNetSocketNewListenTCP(const char *nodename,
fd = -1;
}
- if (nsocks == 0 && familyNotSupported) {
- virReportSystemError(EAFNOSUPPORT, "%s", _("Unable to bind to
port"));
- goto error;
- }
-
- if (nsocks == 0 &&
- addrInUse) {
- virReportSystemError(EADDRINUSE, "%s", _("Unable to bind to
port"));
+ if (nsocks == 0) {
+ if (familyNotSupported)
+ errno = EAFNOSUPPORT;
+ else if(addrInUse)
+ errno = EADDRINUSE;
+ else
+ errno = EDESTADDRREQ;
+ virReportSystemError(errno, "%s", _("Unable to bind to
port"));
goto error;
}
2416
days inactive
2431
days old
6 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Ján Tomko -
Olaf Hering