On 2020-07-20 05:32, Daniel P. Berrange wrote:

On Sun, Jul 19, 2020 at 11:54:06AM +0100, Rui Correia wrote:

...

Greetings folks. I've setup libvirtd on my manjaro linux laptop. Got a couple of VM's running (Win10 and Debian10) through NAT without any issues.

This is what the current network diagram looks like and it works fine:

+-----------------------------------+ | +---------------------+ | | | +----------+ | | | | |Win 10 VM | | | | | |10.1.1.10 | | | | | +----------+ | | | Laptop | | | | Manjaro | +-------------+ | | | 10.0.0.10 | |Debian 10 VM | | | +-------->+ | |10.1.1.11 | | | | | | +-------------+ | | | | |NAT | | | | |10.1.1.0/24 | | | | +---------------------+ | +------------+ | +-----------------------------------+ |router | | |switch +---+ |10.0.0.0/24 | | +---------+ +------------+ | |Desktop | +-------->+Manjaro | |10.0.0.11| +---------+

But now I need the debian machine to be accessible from another host on the lan 10.0.0.0/24 which of course is outside the host. That network diagram would look like this:

+-----------------------------------------+ | +------------------+ | | | +----------+ | | | | |Win 10 VM | | | | | |10.1.1.10 | | | | | NAT +----------+ | | | | 10.1.1.0/24 | | | +------------------+ | | Laptop | +-------->+ Manjaro +------------------------+ | | | 10.0.0.10 | +-------------+ | | | | | |Debian 10 VM | | | | | | |10.2.2.10 | | | | | | Routed +-------------+ | | +------------+ | | | 10.2.2.0/24 | | |router | | | +------------------------+ | |switch +---+ +-----------------------------------------+ |10.0.0.0/24 | | +------------+ | | | +---------+ | |Desktop | +-------->+Manjaro | |10.0.0.11| +---------+

So, I've setup a 'routed network' for the Debian 10 VM but it's not working as I would expect. The host can ping the Debian VM and the Debian VM can ping the host but the Debian VM cannot ping the router 10.0.0.1 or any ip address on the internet. I've been using Virtual Machine Manager to set everything up. And this is how the routed network is configured <network connections="1"> <name>routed</name> <uuid>970a25f7-29b6-4a6b-b890-f593eae4fc15</uuid> <forward dev="wlo1" mode="route"> <interface dev="wlo1"/> </forward> <bridge name="virbr2" stp="on" delay="0"/> <mac address="52:54:00:bf:35:42"/> <domain name="routed"/> <ip address="10.2.2.1" netmask="255.255.255.0"> <dhcp> <range start="10.2.2.2" end="10.2.2.254"/> </dhcp> </ip> </network>

Any idea on what i might be doing wrong?

Is net.ipv4.ip_forward set to 0 ?

He left out, and I should hasten to make clear, it *should* be set to 1 to enable routing: ken@pirouter:/proc/sys/net/ipv4$ cat /proc/sys/net/ipv4/ip_forward 1 And, also, HOLY CROW, you must be an ASCII charting demigod. Did you use software to make those, or do them yourselves? Either way, I'm impressed... -Ken

Hi Daniel, First of all, awfully sorry for replying so late. Unfortunately your reply had gone to the Spam drawer... Also, I'm answering from Gmail's webmail which IIRC only allows for 'quote original post below'. So please forgive me for not following the proper netiquette of 'quote original post above'.

Is net.ipv4.ip_forward set to 0 ?

I assume you're asking if this is setup on the host and not on the VM's. I've checked the host and it is configured like this: $ sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 Should I change it to =0 ? It wouldn't make sense to me if I'd change it to =0 because that way IP Forwarding would be disabled and not enabled as I think enabled is what is needed in this particular case. Also, for this to work fully, your router switch needs to know where to

send traffic for the 10.2.2.1/24 network. IOW, you need to configure static routes on your router switch for 10.2.2.1

I see. So, let me get this straight. What you're saying is that in order for WAN/LAN traffic to reach my VM's I need to set up static IP routes on my router - which is the default gateway for the network - to let it know how to route the packets to the VM's, which needs to be setup to go through the KVM host. Right? But, for testing purposes (trying to reach the VM's from the KVM host) I don't need those static routes, right? Because right now I'd be ok if I could reach the VM's from the KVM host and right now I can't. I've covered a similar setup to yours here:

https://www.berrange.com/posts/2009/12/13/routed-subnets-without-nat-for-lib...

I've looked at your article and I can see that it resembles my setup. But I think I've got that covered. This is what the xml file for the virtual network 'routed' looks like: <network> <name>routed</name> <uuid>970a25f7-29b6-4a6b-b890-f593eae4fc15</uuid> <forward dev="wlo1" mode="route"> <interface dev="wlo1"/> </forward> <bridge name="virbr2" stp="on" delay="0"/> <mac address="52:54:00:bf:35:42"/> <domain name="routed"/> <ip address="10.2.2.1" netmask="255.255.255.0"> <dhcp> <range start="10.2.2.11" end="10.11.22.254"/> </dhcp> </ip> </network> I've got this 'routed' network in use for the virtual nic of my VM's. Weird this is, when I run a net-list command, it comes back empty... $ virsh net-list Name State Autostart Persistent ---------------------------------------- $ This doesn't look ok, right? I mean, running a net-list command it should list all the networks I have set up on this server. But it's coming back empty as if there weren't any networks. I've double checked on Virtual Machine Manager and both the 'default' and 'routed' networks are active and are set up to 'auto start on boot'. Could this somehow be related to my problem? I guess not because the 'default' network is the NAT network that I've used before and it worked and still works when I set the VM's to use it. I'm puzzled with this stuff. Hope you or someone else can put me back on the right track. Thanks in advance. Cheers, Rui Correia On Mon, Jul 20, 2020 at 10:32 AM Daniel P. Berrange <dan@berrange.com> wrote:

...

Greetings folks. I've setup libvirtd on my manjaro linux laptop. Got a couple of VM's running (Win10 and Debian10) through NAT without any issues.

This is what the current network diagram looks like and it works fine:

+-----------------------------------+ | +---------------------+ | | | +----------+ | | | | |Win 10 VM | | | | | |10.1.1.10 | | | | | +----------+ | | | Laptop | | | | Manjaro | +-------------+ | | | 10.0.0.10 | |Debian 10 VM | | | +-------->+ | |10.1.1.11 | | | | | | +-------------+ | | | | |NAT | | | | |10.1.1.0/24 | | | | +---------------------+ | +------------+ | +-----------------------------------+ |router | | |switch +---+ |10.0.0.0/24 | | +---------+ +------------+ | |Desktop | +-------->+Manjaro | |10.0.0.11| +---------+

But now I need the debian machine to be accessible from another host on

...

lan 10.0.0.0/24 which of course is outside the host. That network diagram would look like this:

+-----------------------------------------+ | +------------------+ | | | +----------+ | | | | |Win 10 VM | | | | | |10.1.1.10 | | | | | NAT +----------+ | | | | 10.1.1.0/24 | | | +------------------+ | | Laptop | +-------->+ Manjaro +------------------------+ | | | 10.0.0.10 | +-------------+ | | | | | |Debian 10 VM | | | | | | |10.2.2.10 | | | | | | Routed +-------------+ | | +------------+ | | | 10.2.2.0/24 | | |router | | | +------------------------+ | |switch +---+ +-----------------------------------------+ |10.0.0.0/24 | | +------------+ | | | +---------+ | |Desktop | +-------->+Manjaro | |10.0.0.11| +---------+

So, I've setup a 'routed network' for the Debian 10 VM but it's not working as I would expect. The host can ping the Debian VM and the Debian VM can ping the host but

On Sun, Jul 19, 2020 at 11:54:06AM +0100, Rui Correia wrote: the the

...

Debian VM cannot ping the router 10.0.0.1 or any ip address on the internet. I've been using Virtual Machine Manager to set everything up. And this is how the routed network is configured <network connections="1"> <name>routed</name> <uuid>970a25f7-29b6-4a6b-b890-f593eae4fc15</uuid> <forward dev="wlo1" mode="route"> <interface dev="wlo1"/> </forward> <bridge name="virbr2" stp="on" delay="0"/> <mac address="52:54:00:bf:35:42"/> <domain name="routed"/> <ip address="10.2.2.1" netmask="255.255.255.0"> <dhcp> <range start="10.2.2.2" end="10.2.2.254"/> </dhcp> </ip> </network>

Any idea on what i might be doing wrong?

Is net.ipv4.ip_forward set to 0 ?

Also, for this to work fully, your router switch needs to know where to send traffic for the 10.2.2.1/24 network. IOW, you need to configure static routes on your router switch for 10.2.2.1

I've covered a similar setup to yours here:

https://www.berrange.com/posts/2009/12/13/routed-subnets-without-nat-for-lib...

Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Jul 23, 2020 at 3:54 PM Daniel P. Berrangé <berrange@redhat.com> wrote:

No, don't change it to 0. We need ip_forward enabled as you say.

That's what I thought. I'm leaving it as it is.

Correct. The KVM host knows where the 10.2.2.1/24 subnet is - it owns it. The other hosts on your LAN don't know anything about 10.2.2.1/24, so if they try to access VMs on that subnet, traffic will go to the default route, aka your LAN router. It then has to know which KVM host has the 10.2.2.1/24 subnet to send the traffic onwards.

Yep, again that's what I thought. For now I'll be leaving it as it is because right now I just need the host to be able to communicate with the VM's. Yep, so that suggests a more fundamental problem with the KVM host

config.

Since ip_forward is confirmed as set to 1, I feel the most likely problem is something related to the firewall rules.

Libvirt will create iptables rules to allow traffic. Tradititionally this would have been sufficient, in iptables all rules are in the single set of global tables.

If your OS distro has enabled "nft" to replace iptables though, things become more tricky. In nft world there is no single set of global tables. Any app using nft can define its own top level tables.

So while libvirt adds iptables rules to allow traffic, there is the possibility that "nft" rules may none the less deny traffic.

In the specific case of distros using "firewalld", libvirt does some further workarounds for this problem.

Overall though, I'd be investigating your firewall.

Okay, I think I've understood but how can I tell if my distro has 'nft' enabled? Guess I'll ask down at their IRC channel and see if someone can tell me. Otherwise I think I'm fried because I googled it and I came out empty handed.

It is helpful to add logging rules to your firewall immediately before any REJECT / DROP rules so you can spot packets being dropped. That combined with tcpdump on the TAP devices should help you confirm what is happening to traffic.

I don't have the faintest idea on how to set up logging rules, or worst, how to get a tcpdump on my TAP devices and analyse the dump. I'd use wireshark but I wouldn't know what I'd be doing to analyse the dump with it. Often missed is that there are multiple instances of libvirtd. One global

instance that runs as root for privileged set ups, and then one per user instance that runs unprivileged.

If you run "virsh" as non-root, you'll be querying the per-user instance.

virt-manager uses the privileged instance by default.

Try 'virsh -c qemu:///system netlist' instead, or simply run virsh as root.

Got it. Here's the output with sudo: $ sudo virsh net-list [sudo] password for ******: Name State Autostart Persistent -------------------------------------------- default active yes yes routed active yes yes $ $ virsh -c qemu:///system net-list Name State Autostart Persistent -------------------------------------------- default active yes yes routed active yes yes $ This means both network profiles are created, loaded, active and set up for autostart. Thanks for the headsup. I'll ask the Manjaro guys about the nft. Hopefully they'll know if nft is installed and running. Cheers, Rui Correia

On 7/23/20 6:14 PM, Rui Correia wrote:

On Thu, Jul 23, 2020 at 10:36 PM Rui Correia <rdscorreia74@gmail.com <mailto:rdscorreia74@gmail.com>> wrote:

Thanks for the headsup. I'll ask the Manjaro guys about the nft. Hopefully they'll know if nft is installed and running.

Well, that was fast. I've asked the guys and they told me Manjaro KDE doesn't come with 'nft' installed by default. Then I searched the installed packages by nft and the only thing installed is the libnftnl package which seems to be related to nft but not nft itself. So, I guess my system only has firewall iptables and ufw installed. Hope this helps. I could run wireshark but I wouldn't know what to look for. Any tips?

Back in your original message, you said this: On 7/19/20 6:54 AM, Rui Correia wrote:

The host can ping the Debian VM and the Debian VM can ping the host but the Debian VM cannot ping the router 10.0.0.1 or any ip address on the internet.

But in a later message you say this: On 7/23/20 10:34 AM, Rui Correia wrote:

But, for testing purposes (trying to reach the VM's from the KVM host) I don't need those static routes, right? Because right now I'd be ok if I could reach the VM's from the KVM host and right now I can't.

So which is correct?

<forward dev="wlo1" mode="route"> <interface dev="wlo1"/> </forward>

It will probably make no difference (unless traffic leaving your "KVM Host" isn't actually using the interface named "wlo1", and in that case it makes *all* the difference!), but I would change this to simply: <forward mode='route'/> The purpose of the "forward dev" is commonly misunderstood as having something to do with routine, but it doesn't - it only serves to add an iptables rule that will block traffic if it's coming from or going to any interface other than (in this case) "wlo1". ie. it's a security knob, not a routing knob; if you're not concerned about rogue guests then at best it's just creating extra overhead for each packet, and at worst it could be blocking traffic if it's misconfigured. As for checking with wireshark/tcpdump, mainly the intent is just to see, when you send a packet from one end or the other, whether a corresponding packet shows up in the output of wireshark/tcpdump. As an example, let's say that you are trying to ping (from your original diagram) "desktop manjaro" (10.0.0.11) from "debian 10 VM" (10.2.2.10). First start a ping in a shell on debian 10 VM", then run a command like (as root) this on the KVM Host: tcpdump -i virbr2 -n host 10.2.2.10 You should at least see one icmp "echo request" packet for each ping that is sent. You might even see an icmp response (and if so, hopefully is is an icmp echo reply, rather than destination unreachable or something like that). If you see the outbound icmp echo request and an echo reply, then the problem is on your host or in the guest. If you see an echo request but no echo reply, then look at the next step out - wlo1 interface on the KVM host: tcpdump -i wlo1 -n host 10.2.2.10 You should still see the outbound echo request. If not, then again your problem is on the KVM host. If you see the echo request, but no reply, then you need to go look on "manjaro desktop". Run the same tcpdump command there (as root), but replace "wlo1" with whatever is the name of the ethernet device on that host connecting it to the network. At this point you may see an echo request *and* an outgoing echo response, but not see that response back at the KVM host. That's when you'll want to rerun tcpdump telling it to display the MAC address of the packets: tcpdump -i <whatever-interface-name> -e -n host 10.2.2.10 Now you can look at the MAC address in the tcpdump output - it should contain the MAC of the KVM host, *not* the MAC of your router. If it has the MAC of your router, then you haven't added a routing table entry to the manjaro desktop's network config. Do that. (or, possibly you just want to add a route to the router. That will work, but will result it a lot of duplicated traffic and ICMP redirect packets from the router to the manjaro desktop). Anyway, there are many paths this can take, but that gives you an idea of how to use tcpdump. (you could do the same thing with wireshark, it's just a lot more overhead and lots of info when you really need very little (and also requires that wireshark be installed and a desktop session open, on all the machines involved).