11:38 p.m.
Greetings,
up until a year ago, I was running a server with Debian 10 (stable) on it with the latest
versions of libvirt, qemu and kernel 4.19.x Debian 10 had to offer (both libvirt and qemu
versions were really old).
the network config was simple, one of the vm acted as a router and provided the ip for
both the host and the vm.
I've recently switched distro and now I'm running latest stable libvirt, qemu and
kernel 5.4.x
I've tried to reinstate the network config on the new distro but I cannot get ip via
dhcp for the second vm.
if I assign manual ip and gateway, I have access to the outside world.
here are the relevant dumps:
network on the router vm:
<interface type='network'>
<mac address='52:54:00:53:1c:6b'/>
<source network='default'/>
<target dev='virtsw0-vm1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x21'
slot='0x01' function='0x0'/>
</interface>
the other vm
<interface type='network'>
<mac address='52:54:00:5a:4c:8c'/>
<source network='default'/>
<target dev='virtsw0-vm2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
and finally:
<network connections='2'>
<name>default</name>
<uuid>61bc1a72-bd02-408a-b88e-dec696742c20</uuid>
<bridge name='virtsw0' stp='on' delay='0'/>
<mac address='52:54:00:6b:1b:92'/>
</network>
as it is possible I'm missing a kernel config, here is the output of lsmod:
vfio_pci 49152 6
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 32768 2
vfio 28672 16 vfio_iommu_type1,vfio_pci
ip6table_nat 16384 1
iptable_nat 16384 1
ebtables 24576 0
bridge 143360 0
stp 16384 1 bridge
llc 16384 2 bridge,stp
cfg80211 647168 0
x86_pkg_temp_thermal 20480 0
kvm_intel 237568 8
vhost_net 24576 3
vhost 36864 1 vhost_net
tap 24576 1 vhost_net
kvm 663552 1 kvm_intel
tun 53248 8 vhost_net
r8152 73728 0
nct6775 57344 0
mei_me 32768 0
hwmon_vid 20480 1 nct6775
irqbypass 16384 22 vfio_pci,kvm
mii 16384 1 r8152
mei 77824 1 mei_me
coretemp 16384 0
efivarfs 16384 1
and the .config at https://dpaste.com/9ZUCBDE9R
any ideas how to fix it?
Thanks,
Dagg.
1:22 p.m.
On 9/4/20 12:38 AM, daggs wrote:
Greetings,
up until a year ago, I was running a server with Debian 10 (stable) on it with the latest
versions of libvirt, qemu and kernel 4.19.x Debian 10 had to offer (both libvirt and qemu
versions were really old).
the network config was simple, one of the vm acted as a router and provided the ip for
both the host and the vm.
I've recently switched distro and now I'm running latest stable libvirt, qemu and
kernel 5.4.x
You haven't said which distro, nor what is the libvirt exact libvirt
version (probably won't matter in this case, but in general "libvirt
x.y.z" is more useful than "latest stable libvirt").
If you have full connectivity once you've manually assigned IP
addresses, then you don't have any routing problems, so that can be
counted out. (Anyway, DHCP packets never go beyond the local network).
In that case, you've most likely either got a firewall problem on host
or guest, or a problem with your dhcp server.
I've tried to reinstate the network config on the new distro but
I cannot get ip via dhcp for the second vm.
if I assign manual ip and gateway, I have access to the outside world.
From where? The host? or the guests?
here are the relevant dumps:
network on the router vm:
<interface type='network'>
<mac address='52:54:00:53:1c:6b'/>
<source network='default'/>
<target dev='virtsw0-vm1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x21'
slot='0x01' function='0x0'/>
</interface>
the other vm
<interface type='network'>
<mac address='52:54:00:5a:4c:8c'/>
<source network='default'/>
<target dev='virtsw0-vm2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
and finally:
<network connections='2'>
<name>default</name>
<uuid>61bc1a72-bd02-408a-b88e-dec696742c20</uuid>
<bridge name='virtsw0' stp='on' delay='0'/>
<mac address='52:54:00:6b:1b:92'/>
</network>
Your config is for a bridge that's created by libvirt, but with no
iptables rules, no dnsmasq instance, and no IP address on the host. So
any DHCP server config is outside libvirt's realm, as are any iptables
or nftables rules, so in this case there is nothing to look at in the
libvirt config for either of these issues.
I would start troubleshooting by making sure that the dhcp server is
running, and that you can communicate between the machine with DHCP
server and the guest once a manual IP is assigned. Then use tcpdump or
wireshark at different places on the path between those two to see how
far the DHCP request is getting out, whether a response is being sent by
the server, and if so how far the response is getting back (i.e. on the
host, run tcpdump on the guest's tap device; if you see the DHCP request
there, then run tcpdump on the bridge, if you see it there, run it on
the tap device for the guest, if you see it there, then run tcpdump
inside the guest; then check the dhcp server logs to see if it's
receiving requests. While you're doing all of this, you can also be
noticing whether or not a DHCP response is arriving at each step (and if
you see the response, you can skip looking further ahead in the packet
path, since you know by inference that it made it all the way to the
DHCP server). Once you find the point that the packet is blocked, you'll
be better able to determine why.
as it is possible I'm missing a kernel config, here is the output of lsmod:
vfio_pci 49152 6
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 32768 2
vfio 28672 16 vfio_iommu_type1,vfio_pci
ip6table_nat 16384 1
iptable_nat 16384 1
ebtables 24576 0
bridge 143360 0
stp 16384 1 bridge
llc 16384 2 bridge,stp
cfg80211 647168 0
x86_pkg_temp_thermal 20480 0
kvm_intel 237568 8
vhost_net 24576 3
vhost 36864 1 vhost_net
tap 24576 1 vhost_net
kvm 663552 1 kvm_intel
tun 53248 8 vhost_net
r8152 73728 0
nct6775 57344 0
mei_me 32768 0
hwmon_vid 20480 1 nct6775
irqbypass 16384 22 vfio_pci,kvm
mii 16384 1 r8152
mei 77824 1 mei_me
coretemp 16384 0
efivarfs 16384 1
and the .config at https://dpaste.com/9ZUCBDE9R
any ideas how to fix it?
Thanks,
Dagg.
4:54 p.m.
Greetings Laine,
You haven't said which distro, nor what is the libvirt exact
libvirt
version (probably won't matter in this case, but in general "libvirt
x.y.z" is more useful than "latest stable libvirt").
you are correct, the previous os was debian 10 with libvirt 3, the new os is gentoo with
libvirt 6.2.0
If you have full connectivity once you've manually assigned IP
addresses, then you don't have any routing problems, so that can be
counted out. (Anyway, DHCP packets never go beyond the local network).
In that case, you've most likely either got a firewall problem on host
or guest, or a problem with your dhcp server.
iptables is installed on the host (required by libvirt because of the virt network
features, from what I can see, it isn't running.
the guest is libreelec, somehow I don't think it has iptables installed or
configured.
on the dhcp server (the other vm) I see this:
Sat Sep 5 00:33:25 2020 daemon.info dnsmasq-dhcp[2579]: DHCPDISCOVER(br-lan)
52:54:00:5a:4c:8c
Sat Sep 5 00:33:25 2020 daemon.info dnsmasq-dhcp[2579]: DHCPOFFER(br-lan) 10.0.0.40
52:54:00:5a:4c:8c
multiple times, it means that the server accepted the request and offers the correct ip to
it but doesn't seem to get there.
From where? The host? or the guests?
I can ssh from one vm to another, without the manual ip, I cannot do it
I would start troubleshooting by making sure that the dhcp server is
running, and that you can communicate between the machine with DHCP
server and the guest once a manual IP is assigned. Then use tcpdump or
wireshark at different places on the path between those two to see how
far the DHCP request is getting out, whether a response is being sent by
the server, and if so how far the response is getting back (i.e. on the
host, run tcpdump on the guest's tap device; if you see the DHCP request
there, then run tcpdump on the bridge, if you see it there, run it on
the tap device for the guest, if you see it there, then run tcpdump
inside the guest; then check the dhcp server logs to see if it's
receiving requests. While you're doing all of this, you can also be
noticing whether or not a DHCP response is arriving at each step (and if
you see the response, you can skip looking further ahead in the packet
path, since you know by inference that it made it all the way to the
DHCP server). Once you find the point that the packet is blocked, you'll
be better able to determine why.
alright, I'll try that, thanks.
Dagg.
5:47 p.m.
Greetings Laine,
>
> I would start troubleshooting by making sure that the dhcp server is
> running, and that you can communicate between the machine with DHCP
> server and the guest once a manual IP is assigned. Then use tcpdump or
> wireshark at different places on the path between those two to see how
> far the DHCP request is getting out, whether a response is being sent by
> the server, and if so how far the response is getting back (i.e. on the
> host, run tcpdump on the guest's tap device; if you see the DHCP request
> there, then run tcpdump on the bridge, if you see it there, run it on
> the tap device for the guest, if you see it there, then run tcpdump
> inside the guest; then check the dhcp server logs to see if it's
> receiving requests. While you're doing all of this, you can also be
> noticing whether or not a DHCP response is arriving at each step (and if
> you see the response, you can skip looking further ahead in the packet
> path, since you know by inference that it made it all the way to the
> DHCP server). Once you find the point that the packet is blocked, you'll
> be better able to determine why.
>
>
alright, I'll try that, thanks.
I've ran tcpdump on the vm's tap device, here is what I see:
01:42:15.404754 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from
52:54:00:5a:4c:8c (oui Unknown), length 548
01:42:15.405075 IP Broadcom.Home.bootps > 10.0.0.40.bootpc: BOOTP/DHCP, Reply, length
300
01:42:15.735893 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:6b:1b:92.8003,
length 35
01:42:17.718941 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:6b:1b:92.8003,
length 35
01:42:17.846918 IP6 fe80::fc54:ff:fe5a:4c8c > ff02::2: ICMP6, router solicitation,
length 16
01:42:19.702944 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:6b:1b:92.8003,
length 35
01:42:20.450441 ARP, Request who-has 10.0.0.40 tell Broadcom.Home, length 28
I think that issue is this:
01:42:20.450441 ARP, Request who-has 10.0.0.40 tell Broadcom.Home, length 28
I'm not sure if this is expected but looks like my dhcp server ignores it.
any thoughts on the matter?
Dagg.
2:53 p.m.
On 9/4/20 6:47 PM, daggs wrote:
Greetings Laine,
>>
>> I would start troubleshooting by making sure that the dhcp server is
>> running, and that you can communicate between the machine with DHCP
>> server and the guest once a manual IP is assigned. Then use tcpdump or
>> wireshark at different places on the path between those two to see how
>> far the DHCP request is getting out, whether a response is being sent by
>> the server, and if so how far the response is getting back (i.e. on the
>> host, run tcpdump on the guest's tap device; if you see the DHCP request
>> there, then run tcpdump on the bridge, if you see it there, run it on
>> the tap device for the guest, if you see it there, then run tcpdump
>> inside the guest; then check the dhcp server logs to see if it's
>> receiving requests. While you're doing all of this, you can also be
>> noticing whether or not a DHCP response is arriving at each step (and if
>> you see the response, you can skip looking further ahead in the packet
>> path, since you know by inference that it made it all the way to the
>> DHCP server). Once you find the point that the packet is blocked, you'll
>> be better able to determine why.
>>
>>
>
> alright, I'll try that, thanks.
>
I've ran tcpdump on the vm's tap device, here is what I see:
When you say "the vm", you mean the one running libreelec, that is
trying to get and IP address, correct?
01:42:15.404754 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from 52:54:00:5a:4c:8c (oui Unknown), length 548
01:42:15.405075 IP Broadcom.Home.bootps > 10.0.0.40.bootpc: BOOTP/DHCP, Reply, length
300
I guess Broadcom.home is the IP of the VM that's running the dhcp
server? (I should have suggested using "tcpdump -n -e -v" :-/)
01:42:15.735893 STP 802.1d, Config, Flags [none], bridge-id
8000.52:54:00:6b:1b:92.8003, length 35
01:42:17.718941 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:6b:1b:92.8003,
length 35
01:42:17.846918 IP6 fe80::fc54:ff:fe5a:4c8c > ff02::2: ICMP6, router solicitation,
length 16
01:42:19.702944 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:6b:1b:92.8003,
length 35
01:42:20.450441 ARP, Request who-has 10.0.0.40 tell Broadcom.Home, length 28
I think that issue is this:
01:42:20.450441 ARP, Request who-has 10.0.0.40 tell Broadcom.Home, length 28
I'm not sure if this is expected but looks like my dhcp server ignores it.
any thoughts on the matter?
It looks strange, but is normal. What usually happens is this:
1) The guest sends a DHCP Discover Request, suggesting that it would
like to use the addres 10.0.0.40 (These details will be revealed once
you add "-v" to your tcpdump commandline.
2) The DHCP server says to itself "Hmm, this guy wants to use 10.0.0.40,
which is okay with me, but first I should see if someone else is using
it", so it sends out an ARP request for 10.0.0.40. Then just to be sure,
it sends another.
(at this point, if the server is dnsmasq and it hasn't received an ARP
request, for some reason it sends an ICMP echo request to 10.0.0.40 (the
requested/suggested IP) with destination MAC address of the client that
just sent the DHCP request. No idea why. It won't be answered though
(unless the client actually still had a lease on that address and was
just renewing; but the DHCP server would know it if that's what was
happening, so...)
3) If the server doesn't receive any response to the ARP request, then
it will send a DHCP response to the requested IP + client MAC saying
"Yes, you can use that IP address.
4) I'm not sure why (because it's been > 20 years since I last read the
DHCP RFC), but in the case I just looked at on my host (which is using
dnsmasq as the server, and dhclient as the client), the same request and
response are sent/received at the same IP+MAC addresses a 2nd time.
5) at this point everybody agrees on the new IP address, the client sets
its IP address, the server updates its leases table, and life carries on.
But to back up for a minute - it's completely normal for the DHCP server
to send out an ARP request and get no response. I think things are going
south sometime after that. Are you seeing a DHCP reply at all? If you
don't see it on the libreelec (client) machine's tap device, check if
you see it going *out* on the DHCP server's tap. If it's not there, then
you'll need to debug inside the guest running the DHCP server.
Before this packet is receivd, the guest doesn't yet know that's its IP
address, but it does know that's its MAC address, and it's waiting for a
DHCP reply, so it takes the info from the reply, then sends another
request, this time including all the options it received in the first reply.
4) Now
7:50 a.m.
New subject: KVM network & DHCP
I am very inexperienced with KVM and have a question about networking
As per my basic understanding a network in KVM apparently also provides
the work of an DHCP server for all machines on that network.
So, whether that perception is correct, how would you go about whether
you wanted (for learning purposes) to use a particular VM on such a KVM
(internal) network to act as the DHCP server for all machines on that
network? by default it should conflict with the built in DHCP service of
the network itself, wouldn't it?
8:27 a.m.
New subject: KVM network & DHCP
I looked into this a little more myself (mixture of the "Virtual Machine
Manager" gui and virsh cli is my toolset) which provided me with some
insights.
On 06.09.20 14:50, gunnar.wagner wrote:
/As per my basic understanding a network in KVM apparently also
provides the work of an DHCP server for all machines on that network. /
only if
DHCP is enabled upon creation of such a network
/[...] how would you go about whether you wanted (for learning
purposes) to use a particular VM on such a KVM (internal) network to
act as the DHCP server for all machines on that network?
/
by creating a network without DHCP enabled
11:02 a.m.
Greetings LAine,
When you say "the vm", you mean the one running libreelec,
that is
trying to get and IP address, correct?
yes, you are correct.
I guess Broadcom.home is the IP of the VM that's running the
dhcp
server? (I should have suggested using "tcpdump -n -e -v" :-/)
frankly, I have no idea who is Broadcom.home.
here is the requested dump: https://dpaste.com/849DMX9ND
It looks strange, but is normal. What usually happens is this:
1) The guest sends a DHCP Discover Request, suggesting that it would
like to use the addres 10.0.0.40 (These details will be revealed once
you add "-v" to your tcpdump commandline.
2) The DHCP server says to itself "Hmm, this guy wants to use 10.0.0.40,
which is okay with me, but first I should see if someone else is using
it", so it sends out an ARP request for 10.0.0.40. Then just to be sure,
it sends another.
(at this point, if the server is dnsmasq and it hasn't received an ARP
request, for some reason it sends an ICMP echo request to 10.0.0.40 (the
requested/suggested IP) with destination MAC address of the client that
just sent the DHCP request. No idea why. It won't be answered though
(unless the client actually still had a lease on that address and was
just renewing; but the DHCP server would know it if that's what was
happening, so...)
3) If the server doesn't receive any response to the ARP request, then
it will send a DHCP response to the requested IP + client MAC saying
"Yes, you can use that IP address.
4) I'm not sure why (because it's been > 20 years since I last read the
DHCP RFC), but in the case I just looked at on my host (which is using
dnsmasq as the server, and dhclient as the client), the same request and
response are sent/received at the same IP+MAC addresses a 2nd time.
5) at this point everybody agrees on the new IP address, the client sets
its IP address, the server updates its leases table, and life carries on.
But to back up for a minute - it's completely normal for the DHCP server
to send out an ARP request and get no response. I think things are going
south sometime after that. Are you seeing a DHCP reply at all? If you
don't see it on the libreelec (client) machine's tap device, check if
you see it going *out* on the DHCP server's tap. If it's not there, then
you'll need to debug inside the guest running the DHCP server.
Before this packet is receivd, the guest doesn't yet know that's its IP
address, but it does know that's its MAC address, and it's waiting for a
DHCP reply, so it takes the info from the reply, then sends another
request, this time including all the options it received in the first reply.
4) Now
should I add another nic with static ip and try to trace the pkts from there?
3:38 p.m.
On 9/6/20 12:02 PM, daggs wrote:
You mean so you can ssh to the client/libreelec and run tcpdump there
agains the interface that's doing dhcp? Is tcpdump even available on
libreelec? I know it's very limited, and has no simple facilities for
adding new packages. If it has tcpdump though, then sure. The only
problem is that you would probably not be able to get tcpdump running
via that interface quick enough to see the initial boottime dhcp
exchange; instead you'll probably need to go into the UI and bring the
other interface down/up to trigger a new DHCP cycle.
(BTW, if everything works when the client has a static IP address, then
that proves there is no problem related to ARP requests/responses - that
much is required in order for even a static IP to work)
Greetings LAine,
> When you say "the vm", you mean the one running libreelec, that is
> trying to get and IP address, correct?
yes, you are correct.
> I guess Broadcom.home is the IP of the VM that's running the dhcp
> server? (I should have suggested using "tcpdump -n -e -v" :-/)
>
frankly, I have no idea who is Broadcom.home.
It's just some name tcpdump used to replace the IP address of one of the
machines, and since it's the source IP of a DHCP reply packet, it most
likely is the IP of the DHCP server.
here is the requested dump: https://dpaste.com/849DMX9ND
What I see in that dump is that the DHCP client (Mac address
52:54:00:5a:4c:8c, hostname "streamer" repeatedly sends the exact same
DHCP request (6 times), and the DHCP server responds to each of these
requests alternating between sending the response to the client's MAC
with a destination IP already set, and to the broadcast MAC + IP
addresses) interspersed with several ARP requests directed at the MAC
address of the client asking who has the IP that the server just
suggested (so it's doing something different from what I described in my
previous message - rather than using ARP to verify that an IP isn't
already in use prior to assigning it, it's assuming it has full
authority over IP addresses in the broadcast domain, assigning that IP
to the client without checking for prior use, and then sending the ARP
request to see if the client actually decided to use it.)
Eventually the client gives up (because it hasn't seen any valid DHCP
responses) and gives itself an IP on the 169.254.0.0/16 network, then
goes about the process of looking for other devices to connect to using
that IP.
Was this dump taken on the host of the tap device of the client
(libreelec aka streamer)? If so, I can only see two options: 1) there is
something in iptables or ebtables (or nftables, if you have that on the
host) blocking the DHCP response packets from going out the tap
interface, or 2) there is something in the guest itself blocking the
traffic or preventing the packet from passing.
For (1) you'd need to run "ebtables -L; iptables -S; nft list ruleset"
and look for something suspicious.
For (2) can you try changing both the libreelec and the DHCP server vm's
ethernet device models from virtio to e1000? (or e1000e if they are q35
machinetypes)? If that works, then change one or the other back and see
if it stops working.
should I add another nic with static ip and try to trace the pkts
from there?
7:12 a.m.
Greetings Laine,
It's just some name tcpdump used to replace the IP address of one
of the
machines, and since it's the source IP of a DHCP reply packet, it most
likely is the IP of the DHCP server.
ok, sounds reasonable
> here is the requested dump: https://dpaste.com/849DMX9ND
What I see in that dump is that the DHCP client (Mac address
52:54:00:5a:4c:8c, hostname "streamer" repeatedly sends the exact same
DHCP request (6 times), and the DHCP server responds to each of these
requests alternating between sending the response to the client's MAC
with a destination IP already set, and to the broadcast MAC + IP
addresses) interspersed with several ARP requests directed at the MAC
address of the client asking who has the IP that the server just
suggested (so it's doing something different from what I described in my
previous message - rather than using ARP to verify that an IP isn't
already in use prior to assigning it, it's assuming it has full
authority over IP addresses in the broadcast domain, assigning that IP
to the client without checking for prior use, and then sending the ARP
request to see if the client actually decided to use it.)
Eventually the client gives up (because it hasn't seen any valid DHCP
responses) and gives itself an IP on the 169.254.0.0/16 network, then
goes about the process of looking for other devices to connect to using
that IP.
Was this dump taken on the host of the tap device of the client
(libreelec aka streamer)?
here are the relevant adapters of the vm:
4: virtsw: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group
default qlen 1000
link/ether 52:54:00:6b:1b:92 brd ff:ff:ff:ff:ff:ff
5: virtsw-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virtsw state
DOWN group default qlen 1000
link/ether 52:54:00:6b:1b:92 brd ff:ff:ff:ff:ff:ff
6: nic_host: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UNKNOWN group default qlen 1000
link/ether fe:54:00:a7:79:6b brd ff:ff:ff:ff:ff:ff
inet 11.0.0.3/24 brd 11.0.0.255 scope global dynamic noprefixroute nic_host
valid_lft 33053sec preferred_lft 27653sec
inet6 fdab:9802:eb52::a59/128 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fdab:9802:eb52:0:41d9:d311:10fd:e343/64 scope global mngtmpaddr noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::fc54:ff:fea7:796b/64 scope link
valid_lft forever preferred_lft forever
7: virtsw-router: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master
virtsw state UNKNOWN group default qlen 1000
link/ether fe:54:00:53:1c:6b brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe53:1c6b/64 scope link
valid_lft forever preferred_lft forever
8: virtsw-streamer: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virtsw state UNKNOWN group default qlen 1000
link/ether fe:54:00:5a:4c:8c brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe5a:4c8c/64 scope link
valid_lft forever preferred_lft forever
the dump was taken from the host tapping onto virtsw-streamer.
virtsw-streamer is configured as follows:
<interface type='network'>
<mac address='52:54:00:5a:4c:8c'/>
<source network='default'
portid='77aae31e-5efa-4789-911c-c55b367cd695' bridge='virtsw'/>
<target dev='virtsw-streamer'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
If so, I can only see two options: 1) there is
something in iptables or ebtables (or nftables, if you have that on the
host) blocking the DHCP response packets from going out the tap
interface, or 2) there is something in the guest itself blocking the
traffic or preventing the packet from passing.
For (1) you'd need to run "ebtables -L; iptables -S; nft list ruleset"
and look for something suspicious.
here is what I get:
utils_server /home/igor # ebtables -L; iptables -S; nft list ruleset
The kernel doesn't support the ebtables 'filter' table.
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_FWI -o virtsw -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -i virtsw -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virtsw -o virtsw -j ACCEPT
-A LIBVIRT_INP -i virtsw -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virtsw -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virtsw -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virtsw -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p tcp -m tcp --dport 68 -j ACCEPT
bash: nft: command not found
what about the rest of the ports?
For (2) can you try changing both the libreelec and the DHCP server vm's
ethernet device models from virtio to e1000? (or e1000e if they are q35
machinetypes)? If that works, then change one or the other back and see
if it stops working.
will try and report.
You mean so you can ssh to the client/libreelec and run tcpdump
there
agains the interface that's doing dhcp? Is tcpdump even available on
libreelec? I know it's very limited, and has no simple facilities for
adding new packages. If it has tcpdump though, then sure. The only
problem is that you would probably not be able to get tcpdump running
via that interface quick enough to see the initial boottime dhcp
exchange; instead you'll probably need to go into the UI and bring the
other interface down/up to trigger a new DHCP cycle.
static tcpdump should do the trick imho.
(BTW, if everything works when the client has a static IP address, then
that proves there is no problem related to ARP requests/responses - that
much is required in order for even a static IP to work)
currently I use static ip and I can ssh to the streamer from all machines on the network.
4:47 p.m.
> It's just some name tcpdump used to replace the IP address
of one of the
> machines, and since it's the source IP of a DHCP reply packet, it most
> likely is the IP of the DHCP server.
ok, sounds reasonable
>
> > here is the requested dump: https://dpaste.com/849DMX9ND
>
> What I see in that dump is that the DHCP client (Mac address
> 52:54:00:5a:4c:8c, hostname "streamer" repeatedly sends the exact same
> DHCP request (6 times), and the DHCP server responds to each of these
> requests alternating between sending the response to the client's MAC
> with a destination IP already set, and to the broadcast MAC + IP
> addresses) interspersed with several ARP requests directed at the MAC
> address of the client asking who has the IP that the server just
> suggested (so it's doing something different from what I described in my
> previous message - rather than using ARP to verify that an IP isn't
> already in use prior to assigning it, it's assuming it has full
> authority over IP addresses in the broadcast domain, assigning that IP
> to the client without checking for prior use, and then sending the ARP
> request to see if the client actually decided to use it.)
>
> Eventually the client gives up (because it hasn't seen any valid DHCP
> responses) and gives itself an IP on the 169.254.0.0/16 network, then
> goes about the process of looking for other devices to connect to using
> that IP.
>
> Was this dump taken on the host of the tap device of the client
> (libreelec aka streamer)?
here are the relevant adapters of the vm:
4: virtsw: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group
default qlen 1000
link/ether 52:54:00:6b:1b:92 brd ff:ff:ff:ff:ff:ff
5: virtsw-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virtsw state
DOWN group default qlen 1000
link/ether 52:54:00:6b:1b:92 brd ff:ff:ff:ff:ff:ff
6: nic_host: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UNKNOWN group default qlen 1000
link/ether fe:54:00:a7:79:6b brd ff:ff:ff:ff:ff:ff
inet 11.0.0.3/24 brd 11.0.0.255 scope global dynamic noprefixroute nic_host
valid_lft 33053sec preferred_lft 27653sec
inet6 fdab:9802:eb52::a59/128 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fdab:9802:eb52:0:41d9:d311:10fd:e343/64 scope global mngtmpaddr noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::fc54:ff:fea7:796b/64 scope link
valid_lft forever preferred_lft forever
7: virtsw-router: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virtsw state UNKNOWN group default qlen 1000
link/ether fe:54:00:53:1c:6b brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe53:1c6b/64 scope link
valid_lft forever preferred_lft forever
8: virtsw-streamer: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virtsw state UNKNOWN group default qlen 1000
link/ether fe:54:00:5a:4c:8c brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe5a:4c8c/64 scope link
valid_lft forever preferred_lft forever
the dump was taken from the host tapping onto virtsw-streamer.
virtsw-streamer is configured as follows:
<interface type='network'>
<mac address='52:54:00:5a:4c:8c'/>
<source network='default'
portid='77aae31e-5efa-4789-911c-c55b367cd695' bridge='virtsw'/>
<target dev='virtsw-streamer'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
> If so, I can only see two options: 1) there is
> something in iptables or ebtables (or nftables, if you have that on the
> host) blocking the DHCP response packets from going out the tap
> interface, or 2) there is something in the guest itself blocking the
> traffic or preventing the packet from passing.
>
> For (1) you'd need to run "ebtables -L; iptables -S; nft list
ruleset"
> and look for something suspicious.
here is what I get:
utils_server /home/igor # ebtables -L; iptables -S; nft list ruleset
The kernel doesn't support the ebtables 'filter' table.
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_FWI -o virtsw -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -i virtsw -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virtsw -o virtsw -j ACCEPT
-A LIBVIRT_INP -i virtsw -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virtsw -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virtsw -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virtsw -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virtsw -p tcp -m tcp --dport 68 -j ACCEPT
bash: nft: command not found
what about the rest of the ports?
>
> For (2) can you try changing both the libreelec and the DHCP server vm's
> ethernet device models from virtio to e1000? (or e1000e if they are q35
> machinetypes)? If that works, then change one or the other back and see
> if it stops working.
will try and report.
changed the router's nic type to e1000e and the
streamer's nic type to e1000, error still persists.
> You mean so you can ssh to the client/libreelec and run tcpdump there
> agains the interface that's doing dhcp? Is tcpdump even available on
> libreelec? I know it's very limited, and has no simple facilities for
> adding new packages. If it has tcpdump though, then sure. The only
> problem is that you would probably not be able to get tcpdump running
> via that interface quick enough to see the initial boottime dhcp
> exchange; instead you'll probably need to go into the UI and bring the
> other interface down/up to trigger a new DHCP cycle.
static tcpdump should do the trick imho.
>
> (BTW, if everything works when the client has a static IP address, then
> that proves there is no problem related to ARP requests/responses - that
> much is required in order for even a static IP to work)
currently I use static ip and I can ssh to the streamer from all machines on the
network.
8:56 a.m.
Greetings Laine,
I've found the issue, I was able install tcpdump within libreelec, I saw that the dhcp
sends and receives pkts well.
so I've thought it might be related to the, I've replaced type from virtio to
e1000 and it worked.
libreelec uses kernel 5.1.16 so I assume there is a bug there when it comes to virtio.
I'll use e1000 for now until the next version of libreelec comes out hopefully with
the issue fixed.
thanks for all the help.
Dagg.
1559
days inactive
1571
days old
11 comments
3 participants
participants (3)
-
daggs -
gunnar.wagner -
Laine Stump