6:25 a.m.
Hi,
I hit a libvirt networking problem that guest cannot access host HTTP service. I debug
this issue and tried some efforts. Thanks for your suggestions!
Environment
-------------
guest IP: 192.168.122.46 (Linux, default NAT, installed using virt-manager)
host1 IP: 192.168.3.16 (Centos 8.5 running libvirt and qemu, default libvirt iptable
rules)
HTTP service: 192.168.3.16:70 (firewall rules have allowed this port)
host2: 192.168.3.65:70 (for test only)
guest network xml
<interface type="network">
<mac address="52:54:00:f5:a8:9f"/>
<source network="default"
portid="6e8ce7e7-6517-43fa-b113-aaddb6c1bc08" bridge="virbr0"/>
<target dev="vnet2"/>
<model type="e1000"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00"
slot="0x03" function="0x0"/>
</interface>
1. guest and host1/host2 can ping each other
2. *guest can visit host2 HTTP
3. *guest cannot visit host1 HTTP
When I capture traffic in guest, Wireshark shows:
192.168.122.46 -> 192.168.3.16 // SYN ok
192.168.122.46 <- 192.168.3.16 // Destination unreachable (Port unreachable)
guest route table:
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.122.1 0.0.0.0 UG 100 0 0 eth0
192.168.122.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
host1 route table:
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.3.1 0.0.0.0 UG 100 0 0 enp3s0
192.168.3.0 0.0.0.0 255.255.255.0 U 100 0 0 enp3s0
192.168.122.0 0.0.0.0 255.255.255.0 UG 0 0 0 virbr0
I delete the last rule and add a rule to make sure host1 visits guests will go through
192.168.122.1
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
(other rules omitted)
192.168.122.0 192.168.122.1 255.255.255.0 UG 0 0 0 virbr0
However, traceroute shows this new rule does not work (which should go to 192.168.122.1
first), and guest cannot visit host1 HTTP request.
[!] guest visit http://192.168.3.16:70 does not go through 192.168.122.1
traceroute 192.168.122.46
traceroute to 192.168.122.46 (192.168.122.46), 30 hops max, 60 byte packets
1 192.168.122.46 (192.168.122.46) 0.200 ms 0.194 ms 0.194 ms
# guest visit http://192.168.3.65:70 goes through 192.168.122.1
traceroute 192.168.3.65
traceroute to 192.168.3.65 (192.168.3.65), 30 hops max, 60 byte packets
1 192.168.122.1 (192.168.122.1) 0.244 ms 0.050 ms 0.120 ms
2 192.168.3.65 (192.168.3.65) 0.823 ms !X 0.802 ms !X 0.789 ms !X
In sum, is there a way to force the guest go through 192.168.122.1 when visiting the
hosting machine?
-------------------
libvirt iptable rules:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_INP
-N LIBVIRT_OUT
-N LIBVIRT_FWO
-N LIBVIRT_FWI
-N LIBVIRT_FWX
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable # delete this
rule does not work
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j
ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable # delete this
rule does not work
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT