Cannot start network interfaces
Good day all, I have having an issue starting virtual networks defined within libvirt: virsh # net-start cmdb error: Failed to start network cmdb error: internal error: Child process (VIR_BRIDGE_NAME=virbr3 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/cmdb.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 172.26.80.1: Address already in use The definition of the network is pretty simple: virsh # net-dumpxml cmdb <network> <name>cmdb</name> <uuid>42ffd3c4-624d-46ad-be21-bb3c61c67e41</uuid> <bridge name='virbr3' stp='on' delay='0'/> <mac address='52:54:00:5b:e9:6c'/> <domain name='shihad.org'/> <ip address='172.26.80.1' netmask='255.255.255.0'> </ip> </network> At the OS level, this network interface should get created by libvirt, so I am a bit mystified why dnsmasq is having an issue starting, as there is nothing using this address yet. In fact, virbr3 does not exist until libvirt would bring up this interface: ifconfig -a | grep virbr virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 I am running libvirt 9.0.0 on Debian 12. Any help or pointers greatly appreciated. Thanks! Iain
On Mon, Jan 27, 2025 at 12:50:05PM +0000, Iain M Conochie wrote:
Good day all,
I have having an issue starting virtual networks defined within libvirt:
virsh # net-start cmdb error: Failed to start network cmdb error: internal error: Child process (VIR_BRIDGE_NAME=virbr3 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/cmdb.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 172.26.80.1: Address already in use
The definition of the network is pretty simple:
virsh # net-dumpxml cmdb <network> <name>cmdb</name> <uuid>42ffd3c4-624d-46ad-be21-bb3c61c67e41</uuid> <bridge name='virbr3' stp='on' delay='0'/> <mac address='52:54:00:5b:e9:6c'/> <domain name='shihad.org'/> <ip address='172.26.80.1' netmask='255.255.255.0'> </ip> </network>
At the OS level, this network interface should get created by libvirt, so I am a bit mystified why dnsmasq is having an issue starting, as there is nothing using this address yet. In fact, virbr3 does not exist
There might be, you can probably see an old dnsmasq listening on that address using lsof or ss. This is a bug that is already fixed and it could happen in various ways. This was fixed with a series of patches 97ed0574ea6c..d0a48eeb720f and is included upstream in v10.8.0.
until libvirt would bring up this interface:
ifconfig -a | grep virbr virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
I am running libvirt 9.0.0 on Debian 12.
Unfortunately that either needs to be backported or your system updated. But as a workaround just make sure nothing is listening on that address, which might need killing some dnsmasq processes.
Any help or pointers greatly appreciated.
Thanks!
Iain
On Mon, 2025-01-27 at 14:54 +0100, Martin Kletzander wrote:
On Mon, Jan 27, 2025 at 12:50:05PM +0000, Iain M Conochie wrote:
<snip>
At the OS level, this network interface should get created by libvirt, so I am a bit mystified why dnsmasq is having an issue starting, as there is nothing using this address yet. In fact, virbr3 does not exist
There might be, you can probably see an old dnsmasq listening on that address using lsof or ss. This is a bug that is already fixed and it could happen in various ways.
This was fixed with a series of patches 97ed0574ea6c..d0a48eeb720f and is included upstream in v10.8.0.
until libvirt would bring up this interface:
ifconfig -a | grep virbr virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
I am running libvirt 9.0.0 on Debian 12.
Unfortunately that either needs to be backported or your system updated. But as a workaround just make sure nothing is listening on that address, which might need killing some dnsmasq processes.
I have checked on another machine running the same version of debian, and all the interfaces are down, and no dnsmasq process are running: virsh # net-list --all Name State Autostart Persistent ------------------------------------------------------ ailsa inactive yes yes ailsatech inactive yes yes ansible inactive yes yes chef0 inactive no yes stan inactive yes yes thapxe inactive yes yes thargoid inactive yes yes vagrant-libvirt inactive no yes virsh # net-start ailsa error: Failed to start network ailsa error: internal error: Child process (VIR_BRIDGE_NAME=ailsa /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/ailsa.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 192.168.86.1: Address already in use virsh # quit root@primus:/var/log/libvirt/qemu# ps -ef | grep dnsmas root 105994 102272 0 14:53 pts/1 00:00:00 grep dnsmas I will look into trying to get a later version of libvirt onto my system. Thanks for the pointer Martin! Iain
Any help or pointers greatly appreciated.
Thanks!
Iain
On Mon, Jan 27, 2025 at 02:54:57PM +0000, Iain M Conochie wrote:
On Mon, 2025-01-27 at 14:54 +0100, Martin Kletzander wrote:
On Mon, Jan 27, 2025 at 12:50:05PM +0000, Iain M Conochie wrote:
<snip>
At the OS level, this network interface should get created by libvirt, so I am a bit mystified why dnsmasq is having an issue starting, as there is nothing using this address yet. In fact, virbr3 does not exist
There might be, you can probably see an old dnsmasq listening on that address using lsof or ss. This is a bug that is already fixed and it could happen in various ways.
This was fixed with a series of patches 97ed0574ea6c..d0a48eeb720f and is included upstream in v10.8.0.
until libvirt would bring up this interface:
ifconfig -a | grep virbr virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
I am running libvirt 9.0.0 on Debian 12.
Unfortunately that either needs to be backported or your system updated. But as a workaround just make sure nothing is listening on that address, which might need killing some dnsmasq processes.
I have checked on another machine running the same version of debian, and all the interfaces are down, and no dnsmasq process are running:
virsh # net-list --all Name State Autostart Persistent ------------------------------------------------------ ailsa inactive yes yes ailsatech inactive yes yes ansible inactive yes yes chef0 inactive no yes stan inactive yes yes thapxe inactive yes yes thargoid inactive yes yes vagrant-libvirt inactive no yes
virsh # net-start ailsa error: Failed to start network ailsa error: internal error: Child process (VIR_BRIDGE_NAME=ailsa /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/ailsa.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 192.168.86.1: Address already in use
virsh # quit
root@primus:/var/log/libvirt/qemu# ps -ef | grep dnsmas root 105994 102272 0 14:53 pts/1 00:00:00 grep dnsmas
Try checking what else is using that socket, either with lsof: lsof -i4@192.168.86.1:53 -n or with ss: ss -a sport eq 53 and src eq 192.168.86.1
I will look into trying to get a later version of libvirt onto my system. Thanks for the pointer Martin!
Iain
Any help or pointers greatly appreciated.
Thanks!
Iain
On 1/27/25 05:50, Iain M Conochie wrote:
Good day all,
I have having an issue starting virtual networks defined within libvirt:
virsh # net-start cmdb error: Failed to start network cmdb error: internal error: Child process (VIR_BRIDGE_NAME=virbr3 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/cmdb.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 172.26.80.1: Address already in use
I saw similar reports recently and from the system journal noticed that named started listening on the address as soon as it appeared, before dnsmasq could begin listening. Is the named service running on your system? If so, you could try disabling it and then check if the cmdb network successfully starts. Regards, Jim
On Mon, 2025-01-27 at 15:00 -0700, Jim Fehlig wrote:
On 1/27/25 05:50, Iain M Conochie wrote:
Good day all,
I have having an issue starting virtual networks defined within libvirt:
virsh # net-start cmdb error: Failed to start network cmdb error: internal error: Child process (VIR_BRIDGE_NAME=virbr3 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/cmdb.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 172.26.80.1: Address already in use
I saw similar reports recently and from the system journal noticed that named started listening on the address as soon as it appeared, before dnsmasq could begin listening. Is the named service running on your system? If so, you could try disabling it and then check if the cmdb network successfully starts.
Hi Jim I have locked named to only listen on the primary IP address, and my networks are now starting! Thanks very much for your help. Cheers Iain
Regards, Jim
On 28 Jan 2025 09:20 +0000, from iain@shihad.org (Iain M Conochie):
dnsmasq: failed to create listening socket for 172.26.80.1: Address already in use
I have locked named to only listen on the primary IP address, and my networks are now starting! Thanks very much for your help.
An alternative fix is to $ sudo systemctl disable dnsmasq.service I had what appears to be the same issue after the recent Debian 12.9 upgrade, and that resolved it for me. Not sure if there are side effects to this with accessing VMs by name. -- Michael Kjörling 🔗 https://michael.kjorling.se
On 1/28/25 02:20, Iain M Conochie wrote:
On Mon, 2025-01-27 at 15:00 -0700, Jim Fehlig wrote:
On 1/27/25 05:50, Iain M Conochie wrote:
Good day all,
I have having an issue starting virtual networks defined within libvirt:
virsh # net-start cmdb error: Failed to start network cmdb error: internal error: Child process (VIR_BRIDGE_NAME=virbr3 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/cmdb.conf -- leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper) unexpected exit status 2: dnsmasq: failed to create listening socket for 172.26.80.1: Address already in use
I saw similar reports recently and from the system journal noticed that named started listening on the address as soon as it appeared, before dnsmasq could begin listening. Is the named service running on your system? If so, you could try disabling it and then check if the cmdb network successfully starts.
Hi Jim
I have locked named to only listen on the primary IP address, and my networks are now starting! Thanks very much for your help.
Given several reports across different distros, it would be interesting to know the source of this sudden change in behavior. A change to the bind package? systemd? Something else? Regards, Jim
participants (4)
-
Iain M Conochie -
Jim Fehlig -
Martin Kletzander -
Michael Kjörling