[libvirt] virNetSocketNewListenTCP tries just one address
To rescue this bug from the noise in a subthread: If a hostname resolves to more than one address and the host currently configured itself for just IPv4, doing a bind() to some IPv6 address will fail. As a result an error is returned instead of continuing with the next item in 'runp'.
Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+0000: 4531: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address
After further debugging: 27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35 27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152), inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address) 27672 16:04:46.775455 gettid() = 27672 27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+0000: 27672: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address\n", 132) = 132 27672 16:04:46.775742 gettid() = 27672 27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+0000: 27672: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98 27672 16:04:46.776026 gettid() = 27672 So for some reason libvirtd tries to bind to ipv6 even if the host does not have that ipv6 address at this point, only the link-local address. Not sure if there is a way to detect that within libvirt. Perhaps it should just move on with the runp list and try the next one? Olaf
On Tue, Mar 27, 2018 at 09:57:13AM +0200, Olaf Hering wrote:
To rescue this bug from the noise in a subthread:
If a hostname resolves to more than one address and the host currently configured itself for just IPv4, doing a bind() to some IPv6 address will fail. As a result an error is returned instead of continuing with the next item in 'runp'.
Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+0000: 4531: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address
After further debugging:
27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35 27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152), inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address) 27672 16:04:46.775455 gettid() = 27672 27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+0000: 27672: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address\n", 132) = 132 27672 16:04:46.775742 gettid() = 27672 27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+0000: 27672: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98 27672 16:04:46.776026 gettid() = 27672
So for some reason libvirtd tries to bind to ipv6 even if the host does not have that ipv6 address at this point, only the link-local address. Not sure if there is a way to detect that within libvirt. Perhaps it should just move on with the runp list and try the next one?
It looks like the virNetSocketNewListenTCP only treats EAFNOSUPPORT as an error to continue with - everything else is fatal. At very least we need to add EADDRNOTAVAIL too. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Mar 27, 2018 at 09:57:13AM +0200, Olaf Hering wrote:
To rescue this bug from the noise in a subthread:
If a hostname resolves to more than one address and the host currently configured itself for just IPv4, doing a bind() to some IPv6 address will fail. As a result an error is returned instead of continuing with the next item in 'runp'.
Mär 20 09:35:52 macintyre-old libvirtd[4527]: 2018-03-20 08:35:52.521+0000: 4531: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address
After further debugging:
27672 16:04:46.774906 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 35 27672 16:04:46.775041 setsockopt(35, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 27672 16:04:46.775172 setsockopt(35, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0 27672 16:04:46.775302 bind(35, {sa_family=AF_INET6, sin6_port=htons(49152), inet_pton(AF_INET6, "2620:113:80c0:8000:10:161:8:197", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address) 27672 16:04:46.775455 gettid() = 27672 27672 16:04:46.775590 write(4, "2018-03-20 15:04:46.775+0000: 27672: error : virNetSocketNewListenTCP:389 : Unable to bind to port: Cannot assign requested address\n", 132) = 132 27672 16:04:46.775742 gettid() = 27672 27672 16:04:46.775875 write(4, "2018-03-20 15:04:46.775+0000: 27672: info : virObjectUnref:350 : OBJECT_UNREF: obj=0x7fa4bc003530\n", 98) = 98 27672 16:04:46.776026 gettid() = 27672
So for some reason libvirtd tries to bind to ipv6 even if the host does not have that ipv6 address at this point, only the link-local address. Not sure if there is a way to detect that within libvirt. Perhaps it should just move on with the runp list and try the next one?
The AI_ADDRCONFIG flag to getaddrinfo should have taken care of filtering out IPv6 addresses if you only have the link-local one (fe80::), but 2620:: looks like a legitimate unicast address. Checking for EADDRNOTAVAIL here is intentional to catch configuration errors (i.e. a typo in a literal listen address or misconfigured DNS). Why does your hostname resolve to an unavailable address? Jano
On Tue, Mar 27, Ján Tomko wrote:
Why does your hostname resolve to an unavailable address?
How can the DNS server possibly know how a host has configured itself? In this case I had BOOTPROTO='dhcp4' instead of 'dhcp' in /etc/sysconfig/network/ifcfg-br0 due to all the migration issues I'm seeing. It turned out they are unrelated to such setting. Olaf
On Tue, Mar 27, 2018 at 11:18:10AM +0200, Olaf Hering wrote:
On Tue, Mar 27, Ján Tomko wrote:
Why does your hostname resolve to an unavailable address?
How can the DNS server possibly know how a host has configured itself?
It cannot, but the admin of the network should be able to control both. How can libvirt tell whether this is a misconfiguration of DNS or host's interfaces? But we can possibly apply different rules for different callers: virNetServerServiceNewTCP where silently ignoring a failure for one address on daemon/system startup might be hard to catch and libxlDomainMigrationDstPrepare which was called by an API or by data source - whether it was user provided in the migration URI or libvirtd tried to figure it out For QEMU migration, we allow overriding the default listen address via the migrate_host qemu.conf option, this is passed to QEMU and looking at the code, it goes for best-effort and does not report an error as long as binding to one of the addresses succeeds. Anyway, globally ignoring EADDRNOTAVAIL feels too lenient for me. Jano
In this case I had BOOTPROTO='dhcp4' instead of 'dhcp' in /etc/sysconfig/network/ifcfg-br0 due to all the migration issues I'm seeing. It turned out they are unrelated to such setting.
Olaf
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Mar 27, Ján Tomko wrote:
It cannot, but the admin of the network should be able to control both.
The admin must not control my (test) host, nor must I control the DNS server in the network. But there are likely cases where the admin for DNS and libvirtd is the same person.
How can libvirt tell whether this is a misconfiguration of DNS or host's interfaces?
By simply cycling through the 'runp' list to see if any bind() succeeds? Olaf
On Wed, Mar 28, Olaf Hering wrote:
How can libvirt tell whether this is a misconfiguration of DNS or host's interfaces? By simply cycling through the 'runp' list to see if any bind() succeeds?
This change fixes /etc/sysconfig/network/ifcfg-br0:BOOTPROTO='dhcp4' for me. Just keep going in virNetSocketNewListenTCP: Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.553+0000: 8021: error : virNetSocketNewListenTCP:323 : virNetSocketNewListenTCP: macintyre-old.arch.suse.de 49152 0: Success Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.555+0000: 8021: error : virNetSocketNewListenTCP:394 : virNetSocketNewListenTCP: bind to '10.161.8.197': Ok: Success Apr 11 22:18:02 macintyre-old libvirtd[8017]: 2018-04-11 20:18:02.555+0000: 8021: error : virNetSocketNewListenTCP:394 : virNetSocketNewListenTCP: bind to '2620:113:80c0:8000:10:161:8:197': Ok: Cannot assign requested address You get the idea. Olaf --- src/rpc/virnetsocket.c | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) --- a/src/rpc/virnetsocket.c +++ b/src/rpc/virnetsocket.c @@ -320,6 +320,7 @@ int virNetSocketNewListenTCP(const char *nodename, *retsocks = NULL; *nretsocks = 0; + virReportSystemError(errno, "%s: %s %s %d", __func__, nodename, service, family); memset(&hints, 0, sizeof(hints)); hints.ai_family = family; hints.ai_flags = AI_PASSIVE; @@ -383,12 +384,17 @@ int virNetSocketNewListenTCP(const char *nodename, } } #endif - - if (bind(fd, runp->ai_addr, runp->ai_addrlen) < 0) { - if (errno != EADDRINUSE) { - virReportSystemError(errno, "%s", _("Unable to bind to port")); - goto error; - } + e = bind(fd, runp->ai_addr, runp->ai_addrlen); + { + char hostname[123]; + int r, oe = errno; + memset(hostname, 0, sizeof(hostname)); + r = getnameinfo(runp->ai_addr,runp->ai_addrlen,hostname, sizeof(hostname), NULL, 0, NI_NUMERICHOST); + errno = oe; + virReportSystemError(errno, "%s: bind to '%s': %s", __func__, hostname, r ? gai_strerror(r) : "Ok"); + } + if (e < 0) { + if (errno == EADDRINUSE) addrInUse = true; VIR_FORCE_CLOSE(fd); runp = runp->ai_next; @@ -412,14 +418,14 @@ int virNetSocketNewListenTCP(const char *nodename, fd = -1; } - if (nsocks == 0 && familyNotSupported) { - virReportSystemError(EAFNOSUPPORT, "%s", _("Unable to bind to port")); - goto error; - } - - if (nsocks == 0 && - addrInUse) { - virReportSystemError(EADDRINUSE, "%s", _("Unable to bind to port")); + if (nsocks == 0) { + if (familyNotSupported) + errno = EAFNOSUPPORT; + else if(addrInUse) + errno = EADDRINUSE; + else + errno = EDESTADDRREQ; + virReportSystemError(errno, "%s", _("Unable to bind to port")); goto error; }
participants (3)
-
Daniel P. Berrangé -
Ján Tomko -
Olaf Hering