Hi Peter and other Libvirt-Users,

Thank you, I greatly appreciate your response. The diagram I provided may be a little misleading, so I have attached a new diagram (also available here: https://i.imgur.com/X3nFMCz.png ). I think I also need to provide some more detail about my scenario here:

These physical nodes are provided to me by "bare-metal cloud" providers like Scaleway / OVH / Packet.net / etc.
I have opted to use Ubuntu on each bare-metal server, but I am also well-versed in RHEL/CentOS environments, so if you want to help me out by telling me CentOS-specific details, I'm sure that I can "translate" that to an Ubuntu/Debian setup. Considering my scenario with bare-metal providers, I do not have the flexibility to add/remove physical NICs as I need them, I am sort of "locked in" to a single-IP/single-NIC configuration for each physical node.

But, I *do* have the flexibility of adding more physical nodes as I need them, and I would like to leverage this option when I want to expand my cluster. Its easy for me to buy a bare-metal cloud server (takes a few minutes), and setting up a 1U/2U server by myself on a "lab" network could take me a few hours or so (not to mention, I dont have the resources to host my own network for this). Also, I need to account for the fact that hardware *will* fail somehow eventually, so adding/removing nodes would need to be "easy" in the way that I shouldnt have to configure new DHCP scopes on all the other nodes when I decide to add one more node to the cluster. Also, as a side note, I do not intend to have any central or shared volume pool between the nodes, all storage will be direct on a local LVM volume, and the ability to perform live migrations of guests between hosts is not a high priority for me, I intend for each guest to be "stationary" to each host.

Each node's eth0 is connected via IPSec (unicast and transport mode) to the other. This is to encrypt the communication between all nodes because it travels over the internet. Additionally, each node currently has a vxlan0 interface in the 172.20.0.0/24 network which allows them to be all on the same "LAN". I have attached a simple diagram illustrating this setup, also available here: https://i.imgur.com/jc7bc6b.png . I would prefer to use the DHCP that comes "built-in" to livbirt, but I am open to running DHCP on each host, or as a guest inside each guest network.

So, I hope this sheds a little more light on my scenario.
If you look at the diagram attached, you can see some red lines and some question marks. These red lines are the only part of the environment that needs to be configured, everything else seems to work fine; KVM can provision guests, IPSec tunnel is working, vxlan seems to route/function as I think I would need it, guests can reach the internet, and guests can ping everything else inside the environment, except guests1 cannot ping guests2.

Which makes me question the configuration I have for each guest network.
Each guest network is currently configured with NAT Forwarding, but because KVMHost1 cannot ping guests on KVMHost2, I am concluding that a NAT'd guest network is most likely something that I do not want. I have some choices on how to configure each guest network, it could be an "open" or "routed" network, but I would still need to configure some static routes.

I will take a stab at doing an open or routed network today, but in the meantime, your thoughts and comments about my setup are still welcome and encouraged.

Thank you!
-Cobin



On Wed, May 30, 2018 at 10:44 PM Peter Crowther <peter.crowther@melandra.com> wrote:
I have to say that I wouldn't do the networking that way - in fact, in the clusters I manage, we haven't done the networking that way :-).  Rather than layer 3 routing between VMs, we've chosen to use layer 2 virtual switching (yes, using openvswitch).  We have the luxury of multiple 10G NICs between our hosts, so we've separated out the management network from the guest network, simply to ensure that we retain administrative access to the hosts via ssh.  If you want to live a little more dangerously, you could use VLAN or VXLAN on one NIC - or you could spend a few dollars on an extra network card on each host for the peace of mind!

For the project that's been live for two years: we presently run four hosts on the lab's production network (another two on its acceptance-test network, and another one as a "kickaround" host for playing about with configs).  Guest VMs on all four production hosts share 192.168.59.0/24 (why "59" is a story for another day), on an OVS virtual switch on each host named br-guest, with the guest-specific NIC also set as a port on the virtual switch.  Guest traffic is therefore sent transparently between the hosts where needed, and we can live-migrate a guest from one host to another with no need to change the guest's IP address.  Because we share a common guest network and IP range between all hosts, it's trivial to add (or remove) hosts - no host needs to know anything about routing to another host, and in fact only our management layer cares how many hosts we have.

We happen to have "controller" nodes that run redundant DHCP servers with non-overlapping scopes, but the exact location is not a requirement of this setup.  We could equally well set up a DHCP service on the guest network on each host, allowing allocation of e.g. 192.168.59.1 to .100 on one host, .101 to .200 on another host.  Guests will typically receive offers from each DHCP server and can choose, which is fine as they're all on the same network.  This provides redundancy in case of a full or failed DHCP server, which your routed network approach wouldn't without some careful DHCP forwarding work.

We happen to base our hosts on CentOS 7, but I manage other Debian-derived systems and can probably remember enough about its network setup to help with Ubuntu.  Certainly I can help with OVS weirdnesses; it took some time to get my head round exactly how it works.  That said, I've never set up a kvm host on Debian.

Good luck; happy to provide further pointers if useful.

Cheers,

- Peter

On 30 May 2018 at 15:32, Cobin Bluth <cbluth@gmail.com> wrote:
Hello Libvirt Users,

I would like to setup a two node bare-metal cluster. I need to guidance on the network configuration. I have attached a small diagram, the same diagram can be seen here: https://i.imgur.com/SOk6a6G.png

I would like to configure the following details:
- Each node has a DHCP enabled guest network where VMs will run. (eg, 192.168.1.0/24 for Host1, and 192.168.2.0/24 for Host2)
- Any guest in Host1 should be able to ping guests in Host2, and vice versa.
- All guests have routes to reach the open internet (so that 'yum update' will work "out-of-the-box")
- Each node will be able to operate fully if the other physical node fails. (no central DHCP server, etc)
- I would like to add more physical nodes later when I need the resources.

This is what I have done so far:
- Installed latest Ubuntu 18.04, with latest version of libvirt and supporting software from ubuntu's apt repo.
- Each node can reach the other via its own eth0.
- Each node has a working vxlan0, which can ping the other via its vxlan0, so it looks like the vxlan config is working. (I used ip link add vxlan0 type vxlan...)
- Configured route on Host1 like so: ip route add 192.168.2.0/24 via 172.20.0.1
- Configured route on Host2 also: ip route add 192.168.1.0/24 via 172.20.0.2
- All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and vice versa, yay.
- Guests on Host1 cannot ping guests on Host2, I suspect because the the default NAT config of the libvirt network.

So, at this point I started to search for tutorials or more information/documentation, but I am a little overwhelmed by the sheer amount of information, as well as a lot of "stale" information on blogs etc.
I have learned that I can virsh net-edit default, and then change it to an "open" network: <forward mode='open'/>
After doing this, the guests cannot reach outside their own network, nor reach the internet, so I assume that I would need to add some routes, or something else to get the network functioning like I want it. There is also <forward mode="route"/>, but I dont fully understand the scenarios where one would need an open or a route forward mode. I have also shied away from using openvswitch, and have opted for ifupdown2.
(I have taken most of my inspiration from this blog post: https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/ )

Some questions that I have for the mailing list, any help would be greatly appreciated:
- Is my target configuration of a KVM cluster uncommon? Do you see drawbacks of this setup, or does it go against "typical convention"?
- Would my scenario be better suited for an "open" network or a "route" network?
- What would be the approach to complete this setup?




_______________________________________________
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users