On Thu, Apr 17, 2025 at 03:25:11AM -0000, icefrog1950(a)gmail.com wrote:
Hi,
I hit a libvirt networking problem that guest cannot access host HTTP service. I debug
this issue and tried some efforts. Thanks for your suggestions!
Environment
-------------
guest IP: 192.168.122.46 (Linux, default NAT, installed using virt-manager)
host1 IP: 192.168.3.16 (Centos 8.5 running libvirt and qemu, default libvirt iptable
rules)
HTTP service: 192.168.3.16:70 (firewall rules have allowed this port)
host2: 192.168.3.65:70 (for test only)
guest network xml
<interface type="network">
<mac address="52:54:00:f5:a8:9f"/>
<source network="default"
portid="6e8ce7e7-6517-43fa-b113-aaddb6c1bc08" bridge="virbr0"/>
<target dev="vnet2"/>
<model type="e1000"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x03" function="0x0"/>
</interface>
1. guest and host1/host2 can ping each other
2. *guest can visit host2 HTTP
3. *guest cannot visit host1 HTTP
When I capture traffic in guest, Wireshark shows:
192.168.122.46 -> 192.168.3.16 // SYN ok
192.168.122.46 <- 192.168.3.16 // Destination unreachable (Port unreachable)
guest route table:
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.122.1 0.0.0.0 UG 100 0 0 eth0
192.168.122.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
host1 route table:
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.3.1 0.0.0.0 UG 100 0 0 enp3s0
192.168.3.0 0.0.0.0 255.255.255.0 U 100 0 0 enp3s0
192.168.122.0 0.0.0.0 255.255.255.0 UG 0 0 0 virbr0
I delete the last rule and add a rule to make sure host1 visits guests will go through
192.168.122.1
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
(other rules omitted)
192.168.122.0 192.168.122.1 255.255.255.0 UG 0 0 0 virbr0
However, traceroute shows this new rule does not work (which should go to 192.168.122.1
first), and guest cannot visit host1 HTTP request.
[!] guest visit
http://192.168.3.16:70 does not go through 192.168.122.1
traceroute 192.168.122.46
traceroute to 192.168.122.46 (192.168.122.46), 30 hops max, 60 byte packets
1 192.168.122.46 (192.168.122.46) 0.200 ms 0.194 ms 0.194 ms
# guest visit
http://192.168.3.65:70 goes through 192.168.122.1
traceroute 192.168.3.65
traceroute to 192.168.3.65 (192.168.3.65), 30 hops max, 60 byte packets
1 192.168.122.1 (192.168.122.1) 0.244 ms 0.050 ms 0.120 ms
2 192.168.3.65 (192.168.3.65) 0.823 ms !X 0.802 ms !X 0.789 ms !X
In sum, is there a way to force the guest go through 192.168.122.1 when visiting the
hosting machine?
The information is a bit confusing to me and I might not have the right
answer, but AFAIU the guest is going through the gateway when accessing
the machine, the routing table shows it has nowhere else to go when
going outside the 192.168.122.0/24 network. So either do not use the
outside IP, but 192.168.122.1 _or_ check the rp_filter sysctl for your
interfaces as that seems like it could do something related to that.
-------------------
libvirt iptable rules:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_INP
-N LIBVIRT_OUT
-N LIBVIRT_FWO
-N LIBVIRT_FWI
-N LIBVIRT_FWX
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable # delete this
rule does not work
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j
ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable # delete this
rule does not work
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT