Hello Libvirt Users, I would like to setup a two node bare-metal cluster. I need to guidance on the network configuration. I have attached a small diagram, the same diagram can be seen here: https://i.imgur.com/SOk6a6G.png I would like to configure the following details: - Each node has a DHCP enabled guest network where VMs will run. (eg, *192.168.1.0/24 <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24 <http://192.168.2.0/24>* for Host2) - Any guest in Host1 should be able to ping guests in Host2, and vice versa. - All guests have routes to reach the open internet (so that '*yum update*' will work "out-of-the-box") - Each node will be able to operate fully if the other physical node fails. (no central DHCP server, etc) - I would like to *add more physical nodes later* when I need the resources. This is what I have done so far: - Installed latest Ubuntu 18.04, with latest version of libvirt and supporting software from ubuntu's apt repo. - Each node can reach the other via its own eth0. - Each node has a working vxlan0, which can ping the other via its vxlan0, so it looks like the vxlan config is working. (I used *ip link add vxlan0 type vxlan...*) - Configured route on Host1 like so: *ip route add 192.168.2.0/24 <http://192.168.2.0/24> via 172.20.0.1* - Configured route on Host2 also: *ip route add 192.168.1.0/24 <http://192.168.1.0/24> via 172.20.0.2* - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and vice versa, yay. - Guests on Host1 *cannot* ping guests on Host2, I suspect because the the default NAT config of the libvirt network. So, at this point I started to search for tutorials or more information/documentation, but I am a little overwhelmed by the sheer amount of information, as well as a lot of "stale" information on blogs etc. I have learned that I can *virsh net-edit default*, and then change it to an "open" network:* <forward mode='open'/>* After doing this, the guests cannot reach outside their own network, nor reach the internet, so I assume that I would need to add some routes, or something else to get the network functioning like I want it. There is also *<forward mode="route"/>*, but I dont fully understand the scenarios where one would need an *open* or a *route* forward mode. I have also shied away from using openvswitch, and have opted for ifupdown2. (I have taken most of my inspiration from this blog post: https://joejulian.name/post/how-to-configure-linux- vxlans-with-multiple-unicast-endpoints/ ) Some questions that I have for the mailing list, any help would be greatly appreciated: - Is my target configuration of a KVM cluster uncommon? Do you see drawbacks of this setup, or does it go against "typical convention"? - Would my scenario be better suited for an "*open*" network or a "*route*" network? - What would be the approach to complete this setup? _______________________________________________ libvirt-users mailing list libvirt-users(a)redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users

Hi Peter and other Libvirt-Users, Thank you, I greatly appreciate your response. The diagram I provided may be a little misleading, so I have attached a new diagram (also available here: https://i.imgur.com/X3nFMCz.png ). I think I also need to provide some more detail about my scenario here: These physical nodes are provided to me by "bare-metal cloud" providers like Scaleway / OVH / Packet.net / etc. I have opted to use Ubuntu on each bare-metal server, but I am also well-versed in RHEL/CentOS environments, so if you want to help me out by telling me CentOS-specific details, I'm sure that I can "translate" that to an Ubuntu/Debian setup. Considering my scenario with bare-metal providers, I do *not* have the flexibility to add/remove physical NICs as I need them, I am sort of "locked in" to a single-IP/single-NIC configuration for each physical node. But, I **do** have the flexibility of adding more physical nodes as I need them, and I would like to leverage this option when I want to expand my cluster. Its easy for me to buy a bare-metal cloud server (takes a few minutes), and setting up a 1U/2U server by myself on a "lab" network could take me a few hours or so (not to mention, I dont have the resources to host my own network for this). Also, I need to account for the fact that hardware *will* fail somehow eventually, so adding/removing nodes would need to be "easy" in the way that I shouldnt have to configure new DHCP scopes on all the other nodes when I decide to add one more node to the cluster. Also, as a side note, I do not intend to have any central or shared volume pool between the nodes, all storage will be direct on a local LVM volume, and the ability to perform live migrations of guests between hosts is not a high priority for me, I intend for each guest to be *"stationary" to each host*. Each node's *eth0* is connected via IPSec (unicast and transport mode) to the other. This is to encrypt the communication between all nodes because it travels over the internet. Additionally, each node currently has a *vxlan0* interface in the *172.20.0.0/24 <http://172.20.0.0/24>* network which allows them to be all on the same "LAN". I have attached a simple diagram illustrating this setup, also available here: https://i.imgur.com/jc7bc6b.png . I would *prefer* to use the DHCP that comes "built-in" to livbirt, but I am open to running DHCP on each host, or as a guest inside each guest network. So, I hope this sheds a little more light on my scenario. If you look at the diagram attached, you can see some *red* lines and some question marks. These red lines are the only part of the environment that needs to be configured, everything else seems to work fine; KVM can provision guests, IPSec tunnel is working, vxlan seems to route/function as I think I would need it, guests can reach the internet, and guests can ping everything else inside the environment, *except* guests1 cannot ping guests2. Which makes me question the configuration I have for each guest network. Each guest network is currently configured with NAT Forwarding, but because KVMHost1 cannot ping guests on KVMHost2, I am concluding that a NAT'd guest network is most likely something that I do not want. I have some choices on how to configure each guest network, it could be an "open <https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html>... or "routed <https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode>" network, but I would still need to configure some static routes. I will take a stab at doing an open or routed network today, but in the meantime, your thoughts and comments about my setup are still welcome and encouraged. Thank you! -Cobin On Wed, May 30, 2018 at 10:44 PM Peter Crowther < peter.crowther(a)melandra.com&gt; wrote: > I have to say that I wouldn't do the networking that way - in fact, in > the clusters I manage, we haven't done the networking that way :-). Rather > than layer 3 routing between VMs, we've chosen to use layer 2 virtual > switching (yes, using openvswitch). We have the luxury of multiple 10G > NICs between our hosts, so we've separated out the management network from > the guest network, simply to ensure that we retain administrative access to > the hosts via ssh. If you want to live a little more dangerously, you > could use VLAN or VXLAN on one NIC - or you could spend a few dollars on an > extra network card on each host for the peace of mind! > > For the project that's been live for two years: we presently run four > hosts on the lab's production network (another two on its acceptance-test > network, and another one as a "kickaround" host for playing about with > configs). Guest VMs on all four production hosts share 192.168.59.0/24 > (why "59" is a story for another day), on an OVS virtual switch on each > host named br-guest, with the guest-specific NIC also set as a port on the > virtual switch. Guest traffic is therefore sent transparently between the > hosts where needed, and we can live-migrate a guest from one host to > another with no need to change the guest's IP address. Because we share a > common guest network and IP range between all hosts, it's trivial to add > (or remove) hosts - no host needs to know anything about routing to another > host, and in fact only our management layer cares how many hosts we have. > > We happen to have "controller" nodes that run redundant DHCP servers with > non-overlapping scopes, but the exact location is not a requirement of this > setup. We could equally well set up a DHCP service on the guest network on > each host, allowing allocation of e.g. 192.168.59.1 to .100 on one host, > .101 to .200 on another host. Guests will typically receive offers from > each DHCP server and can choose, which is fine as they're all on the same > network. This provides redundancy in case of a full or failed DHCP server, > which your routed network approach wouldn't without some careful DHCP > forwarding work. > > We happen to base our hosts on CentOS 7, but I manage other > Debian-derived systems and can probably remember enough about its network > setup to help with Ubuntu. Certainly I can help with OVS weirdnesses; it > took some time to get my head round exactly how it works. That said, I've > never set up a kvm host on Debian. > > Good luck; happy to provide further pointers if useful. > > Cheers, > > - Peter > > On 30 May 2018 at 15:32, Cobin Bluth <cbluth(a)gmail.com&gt; wrote: > >> Hello Libvirt Users, >> >> I would like to setup a two node bare-metal cluster. I need to guidance >> on the network configuration. I have attached a small diagram, the same >> diagram can be seen here: https://i.imgur.com/SOk6a6G.png >> >> I would like to configure the following details: >> - Each node has a DHCP enabled guest network where VMs will run. (eg, *192.168.1.0/24 >> <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24 >> <http://192.168.2.0/24>* for Host2) >> - Any guest in Host1 should be able to ping guests in Host2, and vice >> versa. >> - All guests have routes to reach the open internet (so that '*yum >> update*' will work "out-of-the-box") >> - Each node will be able to operate fully if the other physical node >> fails. (no central DHCP server, etc) >> - I would like to *add more physical nodes later* when I need the >> resources. >> >> This is what I have done so far: >> - Installed latest Ubuntu 18.04, with latest version of libvirt and >> supporting software from ubuntu's apt repo. >> - Each node can reach the other via its own eth0. >> - Each node has a working vxlan0, which can ping the other via its >> vxlan0, so it looks like the vxlan config is working. (I used *ip link >> add vxlan0 type vxlan...*) >> - Configured route on Host1 like so: *ip route add 192.168.2.0/24 >> <http://192.168.2.0/24> via 172.20.0.1* >> - Configured route on Host2 also: *ip route add 192.168.1.0/24 >> <http://192.168.1.0/24> via 172.20.0.2* >> - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and >> vice versa, yay. >> - Guests on Host1 *cannot* ping guests on Host2, I suspect because the >> the default NAT config of the libvirt network. >> >> So, at this point I started to search for tutorials or more >> information/documentation, but I am a little overwhelmed by the sheer >> amount of information, as well as a lot of "stale" information on blogs etc. >> I have learned that I can *virsh net-edit default*, and then change it >> to an "open" network:* <forward mode='open'/>* >> After doing this, the guests cannot reach outside their own network, nor >> reach the internet, so I assume that I would need to add some routes, or >> something else to get the network functioning like I want it. There is also *<forward >> mode="route"/>*, but I dont fully understand the scenarios where one >> would need an *open* or a *route* forward mode. I have also shied away >> from using openvswitch, and have opted for ifupdown2. >> (I have taken most of my inspiration from this blog post: >> https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-u... >> ) >> >> Some questions that I have for the mailing list, any help would be >> greatly appreciated: >> - Is my target configuration of a KVM cluster uncommon? Do you see >> drawbacks of this setup, or does it go against "typical convention"? >> - Would my scenario be better suited for an "*open*" network or a " >> *route*" network? >> - What would be the approach to complete this setup? >> >> >> >> >> _______________________________________________ >> libvirt-users mailing list >> libvirt-users(a)redhat.com >> https://www.redhat.com/mailman/listinfo/libvirt-users >> > >