Re: FreeBSD dhcp failing with UDP checksum errors

On Mon, Oct 14, 2024 at 04:55:37PM +0100, Daniel P. Berrangé wrote: > On Mon, Oct 14, 2024 at 04:37:42PM +0100, Richard W.M. Jones wrote: >> On Mon, Oct 14, 2024 at 10:46:22AM -0400, Laine Stump wrote: >>> On 10/14/24 5:35 AM, Richard W.M. Jones wrote: >>>> On Mon, Oct 14, 2024 at 09:52:13AM +0100, Daniel P. Berrangé wrote: >>>>> >>>>> Urgh, I wonder if this is fallout from switching to NFT instead of iptables. >>>> >>>> I can list the firewall rules if you tell me what I'm looking for ... >>>> >>>>> IIUC, the NFT kernel maintainers didn't implement for checksum fixup rules, >>>>> since they believe that all modern distros would have long ago fixed their >>>>> bugs wrt mangled checksums. >>> >>> That's the first thing that came to my mind too - maybe RHEL5 >>> *isn't* the only guest OS that has this problem. (I certainly hope >>> that isn't the case :-/) >>> >>> There are two ways to test out this theory: >>> >>> 1) change the setting of "firewall_backend" in >>> /etc/libvirt/network.conf to "iptables" and restart virtnetworkd >>> >>> (if that does work, then switch back to nftables, restart >>> virtnetworkd, and test again just to make sure the issue wasn't >>> caused by some out-of-place rule) >> >> I changed the setting between nftables and iptables a few times and I >> can confirm that your theory seems to be correct. >> >> iptables => >> >> "5 bad udp checksums in 5 packets" message is NOT seen >> >> FreeBSD gets an immediate DHCPOFFER and boots quickly with network >> >> nftables => >> >> FreeBSD sends 5 DHCPDISCOVER messages >> >> "5 bad udp checksums in 5 packets" reappears >> >> FreeBSD does NOT see DHCPOFFER, although it does seem to remember >> the offer from the previous boot, so it does get a network >> connection in the end. >> >>> or >>> >>> 2) tell qemu to setup the virtio-net device to do its packet >>> processing in userspace rather than the kernel. You do this by >>> adding >>> >>> <driver name='qemu'/> >>> >>> to the <interface> section. >> >> This also works (with nftables). >> >>>> If I understand the trace correctly, the bad checksum originates on >>>> the Linux host (the reply sent by dnsmasq). >>> >>> I need to try it again to verify, but my recollection is that (when >>> you're using virtio-net with default settings) the checksums of DHCP >>> packets in one direction or the other *always* show up in tcpdump as >>> having bad checksums, but they still end up getting to the other end >>> with a proper checksum. Sometime in the distant past I *may have* >>> had it explained to me why this happens, but I don't recall now. >>> Anyway, I'm just saying this so that you know the validity of the >>> UDP checksum shouldn't be used as an indicator of whether or not >>> things are "working". >> >> I have to say I also don't really understand what's happening here. >> Isn't the Linux host sending DHCPOFFER? Why doesn't it set the UDP >> checksum correctly and/or why would tcpdump report it wrongly if it is >> setting it? > > Here are the original gory details > > https://lists.isc.org/pipermail/dhcp-hackers/2010-April/001835.html > > TL;DR: we have checksum offload running so the host doesn't fill > in any checksums, but DHCP client then tries to validate the > non-existant checksum. Boom. ISC DHCP fixed this in https://github.com/isc-projects/dhcp/commit/7ff6ae5aa85754119319def3c7f22... and if i'm interpreting this patch correctly, it is only fixed on Linux - most changes are in lpf.c, which is "Linux Packet Filter", and I'm assumnig that codepath won't be used on *BSD. If correct, then the idea that checksum fixup from iptables is obsolete is incorrect, and we need it added to nftables for parity. Requiring users to turn off vhost-net feature is horrible, not just for the user experiance of not having a broken VM out of the box, but also for performance, as checksum offloading is a good thing if you want fast networking.