Hi, I'm not sure what sort of diagram you mean, but I'll have a try. Does this help? It sounds like I need to do something to enable routing on what I've labelled "NIC3" on the diagram - can you please explain what I need to do? [image: Inline image 1] Regards, Phill On Tue, Sep 1, 2015 at 4:53 PM, Ajey Gore <ajeygore@gmail.com> wrote:

Can you please drop a rough diagram here? I think you are routing through this VM and must have shared the host interface.

- ajey

On Tue, Sep 1, 2015 at 7:39 AM, Phill Edwards <philledwards@gmail.com> wrote:

...

I'm pretty new to KVM and have a KVM CentOS 7.1 hypervisor running a few VMs. I'm moving all my VMs from an ESXi host as I want to use KVM in future. Most of my VMs are working except for one which is running a Sophos UTM router (Sophos UTM is similar to products like pfSense <https://www.pfsense.org/>, Smoothwall <http://www.smoothwall.org/> etc).

The host has 3 physical NICs which are configured on the Sophos VM as: 1) LAN (fixed IP) 2) DMZ (fixed IP) 3) WAN (which is directly plugged into a cable modem for the internet connection and is configured DHCP).

I have imported the settings from the "old" Sophos machine so I know the configuration of the new one is identical to the old one. I have even tried configuring the NICs to have the same MAC addresses as the old one.

The problem is that no matter what I try I cannot get the WAN NIC to get an internet link up and running with my cable modem. I have re-installed the VM countless times, turned off the modem and VM, done a factory reset of the modem, and, as I mentioned, ensured the MAC addresses are the same. Nothing I try has been successful.

The network interfaces on the new Sophos VM look like this: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:de brd ff:ff:ff:ff:ff:ff inet 192.168.0.254/24 brd 192.168.0.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:e8 brd ff:ff:ff:ff:ff:ff 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:f2 brd ff:ff:ff:ff:ff:ff inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2 valid_lft forever preferred_lft forever

I also found this in the /var/log/system.log of the Sophos VM: 2015:08:29-12:04:05 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 6 2015:08:29-12:04:11 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 13 2015:08:29-12:04:24 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 2 2015:08:29-12:04:26 sop dhclient: No DHCPOFFERS received.

I have shut down firewalld on the KVM host so I don't think there are any firewall rules blocking this.

As soon as I fire up the original Sophos VM on ESXi the internet connection works perfectly again.

If I can't get this VM running on KVM it's a show-stopper. Can anyone suggest what might be going on that is preventing the WAN link from connecting? Or suggest a way of troubleshooting this?

Thanks in advance.

Hi Ajey, I thought I already was doing bridge mode. Below are the network interface definitions from the XML config file for the Sophos VM. (Note that it's actually the middle definition which is connected to the cable modem which is different to how I showed it in the earlier diagram). What I don't understand is that the interface type says "direct" in the XML even though in virt-manager it shows up as "Bridge": [image: Inline image 1] <interface type='bridge'> <mac address='52:54:00:63:2e:15'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='direct'> <mac address='00:0c:29:79:d4:e8'/> <source dev='enp5s0f0' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <interface type='direct'> <mac address='52:54:00:42:33:92'/> <source dev='enp5s0f1' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> Regards, Phill On Tue, Sep 1, 2015 at 10:25 PM, Ajey Gore <ajeygore@gmail.com> wrote:

does you cable modem give you automatically the IP address? if thats the case then you need to do bridge configuration between nic3 and macvtap dhcp ip

and then you will be able to get dhcp attached IP to sophos vm

- ajey

On Tue, Sep 1, 2015 at 4:33 PM, Phill Edwards <philledwards@gmail.com> wrote:

...

Hi, I'm not sure what sort of diagram you mean, but I'll have a try. Does this help? It sounds like I need to do something to enable routing on what I've labelled "NIC3" on the diagram - can you please explain what I need to do?

<image.png>

Regards, Phill

On Tue, Sep 1, 2015 at 4:53 PM, Ajey Gore <ajeygore@gmail.com> wrote:

...

Can you please drop a rough diagram here? I think you are routing through this VM and must have shared the host interface.

- ajey

On Tue, Sep 1, 2015 at 7:39 AM, Phill Edwards <philledwards@gmail.com> wrote:

...

I'm pretty new to KVM and have a KVM CentOS 7.1 hypervisor running a few VMs. I'm moving all my VMs from an ESXi host as I want to use KVM in future. Most of my VMs are working except for one which is running a Sophos UTM router (Sophos UTM is similar to products like pfSense <https://www.pfsense.org/>, Smoothwall <http://www.smoothwall.org/> etc).

The host has 3 physical NICs which are configured on the Sophos VM as: 1) LAN (fixed IP) 2) DMZ (fixed IP) 3) WAN (which is directly plugged into a cable modem for the internet connection and is configured DHCP).

I have imported the settings from the "old" Sophos machine so I know the configuration of the new one is identical to the old one. I have even tried configuring the NICs to have the same MAC addresses as the old one.

The problem is that no matter what I try I cannot get the WAN NIC to get an internet link up and running with my cable modem. I have re-installed the VM countless times, turned off the modem and VM, done a factory reset of the modem, and, as I mentioned, ensured the MAC addresses are the same. Nothing I try has been successful.

The network interfaces on the new Sophos VM look like this: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:de brd ff:ff:ff:ff:ff:ff inet 192.168.0.254/24 brd 192.168.0.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:e8 brd ff:ff:ff:ff:ff:ff 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:f2 brd ff:ff:ff:ff:ff:ff inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2 valid_lft forever preferred_lft forever

I also found this in the /var/log/system.log of the Sophos VM: 2015:08:29-12:04:05 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 6 2015:08:29-12:04:11 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 13 2015:08:29-12:04:24 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 2 2015:08:29-12:04:26 sop dhclient: No DHCPOFFERS received.

I have shut down firewalld on the KVM host so I don't think there are any firewall rules blocking this.

As soon as I fire up the original Sophos VM on ESXi the internet connection works perfectly again.

If I can't get this VM running on KVM it's a show-stopper. Can anyone suggest what might be going on that is preventing the WAN link from connecting? Or suggest a way of troubleshooting this?

Thanks in advance.

Before anything else, a request - please don't top-post responses on technical lists. It is much easier to follow the details of the conversation if you respond inline with the part of the message you are replying to. On 09/01/2015 11:37 AM, Daniel Sanabria wrote:

Hi Phil,

But you said you disabled firewalld which makes me wonder if you have the necessary forwarding rules active.

There is some misinformation/confusion going on here. firewalld manages iptables and ebtables rules. iptables/ebtables rules on the host do not apply to macvtap interfaces (i.e. the data path doesn't even go through iptables or ebtables on the host), so it shouldn't matter what the state of the host's firewall is. Also, there are no "forwarding rules" of any type needed anywhere for a macvtap connection.

Dan

On 1 September 2015 at 13:48, Phill Edwards <philledwards@gmail.com <mailto:philledwards@gmail.com>> wrote:

Hi Ajey, I thought I already was doing bridge mode. Below are the network interface definitions from the XML config file for the Sophos VM. (Note that it's actually the middle definition which is connected to the cable modem which is different to how I showed it in the earlier diagram). What I don't understand is that the interface type says "direct" in the XML even though in virt-manager it shows up as "Bridge":

"direct" means macvtap, and macvtap operates in 4 different modes: * vepa (only applicable when connecting to certain IBM switches) * bridge (multiple guests using the same physdev can communicate directly) * private (multiple guests, but all traffic must go out the physdev and back for inter-guest communication) * passthrough (only one guest is allowed, and it effectively takes over the physdev) So you are using the most permissive mode of macvtap - other guests could connect to that physdev and they sould be able to talk to the router guest (as long as the firewall *inside the guest* It really looks to me, at first glance, that there aren't any problems with the guest's network config. So some random questions/ideas that may or may not lead to better understanding and possibly a solution: 1) Does the cable modem setup NAT in front of a private network, and allow multiple dhcp clients to get an IP address for that network? Or is is a "passthrough" type of device that only responds to the first device that has sent a DHCP request since it was booted? If the latter: a) what is the config of enp5s0f0 on the host? Is it getting an IP address from the cable modem (thus preventing the Sophos guest from getting one)? b) Or possibly have you not rebooted the cable modem after shutting down the ESXi-based guest, and that guest's traffic isn't actually arriving at the cable modem with the 00:0c:.... MAC address, but with the MAC address of the physical interface on the ESXi host? 2) Have you rebooted the cable modem after shutting down the Sophos guest on the ESXi host? Have you shutdown the ESXi host completely? 3) try using tcpdump on the host to monitor traffic on the macvtapX (probably macvtap0) device that is created for the guest to see if the dhcp request is actually making it out of the guest. After that, check if it is seen on enp5s0f0 and, if you can, see if the cable modem is receiving it. If all of those are positive, try looking for the DHCP response packets at the same locations (in reverse order) - basically the idea is to see where the break in communications is, then you can drill in to see why that would happen.

<interface type='bridge'> <mac address='52:54:00:63:2e:15'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='direct'> <mac address='00:0c:29:79:d4:e8'/> <source dev='enp5s0f0' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <interface type='direct'> <mac address='52:54:00:42:33:92'/> <source dev='enp5s0f1' mode='bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface>

Regards, Phill

On Tue, Sep 1, 2015 at 10:25 PM, Ajey Gore <ajeygore@gmail.com <mailto:ajeygore@gmail.com>> wrote:

does you cable modem give you automatically the IP address? if thats the case then you need to do bridge configuration between nic3 and macvtap dhcp ip

I'm not sure exactly what you mean by this, but if you're using macvtap, it doesn't matter which mode you are using, all traffic from the physical network should be seen by the guest. The macvtap mode only makes a difference in communication with other guests, and in whether or not the host can simultaneously use the physical device (in passthrough mode, the physdev is completely unusable by the host).

and then you will be able to get dhcp attached IP to sophos vm

- ajey

On Tue, Sep 1, 2015 at 4:33 PM, Phill Edwards <philledwards@gmail.com <mailto:philledwards@gmail.com>> wrote:

Hi, I'm not sure what sort of diagram you mean, but I'll have a try. Does this help? It sounds like I need to do something to enable routing on what I've labelled "NIC3" on the diagram - can you please explain what I need to do?

For future reference, what is most helpful in the first pass of debugging libvirt-related networking problems is the <interface> sections from the domain xml of the guest, and sometimes the output of "ip -d link show" from the host. (The diagram was also helpful, but the first thing I always want to see is the <interface> XML)

<image.png>

Regards, Phill

On Tue, Sep 1, 2015 at 4:53 PM, Ajey Gore <ajeygore@gmail.com <mailto:ajeygore@gmail.com>> wrote:

Can you please drop a rough diagram here? I think you are routing through this VM and must have shared the host interface.

- ajey

On Tue, Sep 1, 2015 at 7:39 AM, Phill Edwards <philledwards@gmail.com <mailto:philledwards@gmail.com>> wrote:

I'm pretty new to KVM and have a KVM CentOS 7.1 hypervisor running a few VMs. I'm moving all my VMs from an ESXi host as I want to use KVM in future. Most of my VMs are working except for one which is running a Sophos UTM router (Sophos UTM is similar to products like pfSense <https://www.pfsense.org/>, Smoothwall <http://www.smoothwall.org/> etc).

The host has 3 physical NICs which are configured on the Sophos VM as: 1) LAN (fixed IP) 2) DMZ (fixed IP) 3) WAN (which is directly plugged into a cable modem for the internet connection and is configured DHCP).

I have imported the settings from the "old" Sophos machine so I know the configuration of the new one is identical to the old one. I have even tried configuring the NICs to have the same MAC addresses as the old one.

The problem is that no matter what I try I cannot get the WAN NIC to get an internet link up and running with my cable modem. I have re-installed the VM countless times, turned off the modem and VM, done a factory reset of the modem, and, as I mentioned, ensured the MAC addresses are the same. Nothing I try has been successful.

The network interfaces on the new Sophos VM look like this: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:de brd ff:ff:ff:ff:ff:ff inet 192.168.0.254/24 <http://192.168.0.254/24> brd 192.168.0.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:e8 brd ff:ff:ff:ff:ff:ff 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:79:d4:f2 brd ff:ff:ff:ff:ff:ff inet 192.168.1.254/24 <http://192.168.1.254/24> brd 192.168.1.255 scope global eth2 valid_lft forever preferred_lft forever

I also found this in the /var/log/system.log of the Sophos VM: 2015:08:29-12:04:05 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 6 2015:08:29-12:04:11 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 13 2015:08:29-12:04:24 sop dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 2 2015:08:29-12:04:26 sop dhclient: No DHCPOFFERS received.

I have shut down firewalld on the KVM host so I don't think there are any firewall rules blocking this.

As soon as I fire up the original Sophos VM on ESXi the internet connection works perfectly again.

If I can't get this VM running on KVM it's a show-stopper. Can anyone suggest what might be going on that is preventing the WAN link from connecting? Or suggest a way of troubleshooting this?

Thanks in advance.