[libvirt] networking with two hosts
The following situation is given: Host A with VM1 and VM2 running. Host B with VM3 und VM4 running. Task 1.) Each VM must have network access to each other VM and to the outside. Task 2.) When one host fails, all VMs are running on the remainig host and 1.) must still be fulfilled. Does somebody has any hints about realising this scenario with libvirt and KVM? mfg ar -- Hier könnte keine Signatur stehen.
On Tue, 2009-02-24 at 14:25 +0100, Andreas Rittershofer wrote:
Task 1.) Each VM must have network access to each other VM and to the outside. Task 2.) When one host fails, all VMs are running on the remainig host and 1.) must still be fulfilled.
Does somebody has any hints about realising this scenario with libvirt and KVM?
What you want is a 'shared network interface'[1] David [1] http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_...
Am 25.02.2009 um 00:11 schrieb David Lutterkort:
On Tue, 2009-02-24 at 14:25 +0100, Andreas Rittershofer wrote:
Task 1.) Each VM must have network access to each other VM and to the outside. Task 2.) When one host fails, all VMs are running on the remainig host and 1.) must still be fulfilled.
Does somebody has any hints about realising this scenario with libvirt and KVM?
What you want is a 'shared network interface'[1]
David
[1] http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_...
Thank you for this answer - but it is too short. I'm working since two days on this problem but I cannot solve it until now. Some configurations are working when the VMs are on different nodes but are failing when all machines are running on one node and vice versa. Can you please go in more details? mfg ar -- Hier könnte keine Signatur stehen.
Andreas Rittershofer napsal(a):
Am 25.02.2009 um 00:11 schrieb David Lutterkort:
On Tue, 2009-02-24 at 14:25 +0100, Andreas Rittershofer wrote:
Task 1.) Each VM must have network access to each other VM and to the outside. Task 2.) When one host fails, all VMs are running on the remainig host and 1.) must still be fulfilled.
Does somebody has any hints about realising this scenario with libvirt and KVM?
What you want is a 'shared network interface'[1]
David
[1] http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_...
Thank you for this answer - but it is too short.
I'm working since two days on this problem but I cannot solve it until now.
Some configurations are working when the VMs are on different nodes but are failing when all machines are running on one node and vice versa.
Can you please go in more details?
mfg ar
Basically the idea is like this. You have your hosts on one ethernet segment. You create bridge on each hosts and put hosts physical adapter into it (lets call the bridges BrA and BrB). The bridge behaves like a virtual ethernet switch. You should configure IP on the bridge not on the physical interface itself. And now if you add any other adapter into the bridge, it will act as if it is directly on the ethernet. This has been used for bridging two or more ethernet segments into one for a long time. If you setup your VMs correctly (see the wiki page for details), libivirt will create hosts virtual adapter and add it to the bridge. For example for VM1 running on A it will create i.e. vnet1 and add it to the bridge BrA. The virtualized interface in the VM1 will have direct access to the ethernet segment. It can have IP address from the same range as the hosts, etc... If the host A fails, then VM1 and VM2 will be down but the VM3 and VM4 wont be affected (besides they can not communicate with VM1 or VM2 of course). However if you (live) migrate i.e. VM1 from A to B, its network interface will be removed from BrA, destroyed, brought up on B and inserted into BrB with the same MAC address. It will also send ARP packet for all switches to update new location of that MAC address. This can be done with very shot downtime and network operations continues on new host without interruption. Do not forget to setup firewall on hosts as it is described in wiki page. Radek
Am 25.02.2009 um 23:28 schrieb Radek Hladik:
Andreas Rittershofer napsal(a):
Am 25.02.2009 um 00:11 schrieb David Lutterkort:
On Tue, 2009-02-24 at 14:25 +0100, Andreas Rittershofer wrote:
Task 1.) Each VM must have network access to each other VM and to the outside. Task 2.) When one host fails, all VMs are running on the remainig host and 1.) must still be fulfilled.
Does somebody has any hints about realising this scenario with libvirt and KVM?
Thank you for your answer.
Basically the idea is like this. You have your hosts on one ethernet segment. You create bridge on each hosts and put hosts physical adapter into it (lets call the bridges BrA and BrB). The bridge behaves like a virtual ethernet switch. You should configure IP on the bridge not on the physical interface itself. And now if you add any other adapter into the bridge, it will act as if it is directly on the ethernet. This has been used for bridging two or more ethernet segments into one for a long time.
This was not the problem and runs fine.
If you setup your VMs correctly (see the wiki page for details), libivirt will create hosts virtual adapter and add it to the bridge. For example for VM1 running on A it will create i.e. vnet1 and add it to the bridge BrA. The virtualized interface in the VM1 will have direct access to the ethernet segment. It can have IP address from the same range as the hosts, etc...
This was not the problem too and runs fine too. My problem is: 1.) VM1 running on host A must have network access to VM3 when VM3 is running on host B by default _and_ 2.) when VM3 is running on host A because host B failed. In some of my configurations 1.) worked but 2.) not, in some other configurations 2.) worked but 1.) not. I must have missed some detail ... mfg ar -- Hier könnte keine Signatur stehen.
Andreas Rittershofer napsal(a):
My problem is: 1.) VM1 running on host A must have network access to VM3 when VM3 is running on host B by default _and_ 2.) when VM3 is running on host A because host B failed.
In some of my configurations 1.) worked but 2.) not, in some other configurations 2.) worked but 1.) not.
I must have missed some detail ...
I do not see the problem. With the bridged scenario every VM has direct access to the physical network and it is not important on which host it runs. If the VMs are running on the same host, their ethernet frames will just "turn around" in the bridge interface and if they are running on different hosts they will go thru the real ethernet. State 1) A: VM1 B: VM3 Eth. frames will go like this: VM1->BrA->PhysCardA->Ethernet->PhysCardB->BrB->VM3 State 2) A: VM1+VM3 B: x Eth. frames will go like this: VM1->BrA->VM3 This is how it should work, for more debugging I would suggest running tcpdump on all interfaces and pinging the hosts - tcpdump on bridge may not work correctly (I am not sure about it) but on the other (vnet and eth) interfaces it will work. Did you setup the firewall? I need to do #iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT to allow bridged packets thru the firewall. Radek
Am 26.02.2009 um 20:26 schrieb Radek Hladik:
Andreas Rittershofer napsal(a):
My problem is: 1.) VM1 running on host A must have network access to VM3 when VM3 is running on host B by default _and_ 2.) when VM3 is running on host A because host B failed. In some of my configurations 1.) worked but 2.) not, in some other configurations 2.) worked but 1.) not. I must have missed some detail ...
I do not see the problem. With the bridged scenario every VM has direct access to the physical network and it is not important on which host it runs. If the VMs are running on the same host, their ethernet frames will just "turn around" in the bridge interface and if they are running on different hosts they will go thru the real ethernet.
State 1) A: VM1 B: VM3
Eth. frames will go like this: VM1->BrA->PhysCardA->Ethernet-
PhysCardB->BrB->VM3 State 2)
A: VM1+VM3 B: x
Eth. frames will go like this: VM1->BrA->VM3
NOW I've got it! The errors in my configuration were two different network configurations on host A and on host B - I always thought that the network on host A and the network on host B must have different IP addresses. Now there is only on network configuration for host A _and_ host B and everything is fine! Thank you very much for your help! mfg ar -- Hier könnte keine Signatur stehen.
Andreas Rittershofer napsal(a):
Am 26.02.2009 um 20:26 schrieb Radek Hladik:
Andreas Rittershofer napsal(a):
My problem is: 1.) VM1 running on host A must have network access to VM3 when VM3 is running on host B by default _and_ 2.) when VM3 is running on host A because host B failed. In some of my configurations 1.) worked but 2.) not, in some other configurations 2.) worked but 1.) not. I must have missed some detail ...
I do not see the problem. With the bridged scenario every VM has direct access to the physical network and it is not important on which host it runs. If the VMs are running on the same host, their ethernet frames will just "turn around" in the bridge interface and if they are running on different hosts they will go thru the real ethernet.
State 1) A: VM1 B: VM3
Eth. frames will go like this: VM1->BrA->PhysCardA->Ethernet->PhysCardB->BrB->VM3 State 2)
A: VM1+VM3 B: x
Eth. frames will go like this: VM1->BrA->VM3
NOW I've got it! The errors in my configuration were two different network configurations on host A and on host B - I always thought that the network on host A and the network on host B must have different IP addresses. Now there is only on network configuration for host A _and_ host B and everything is fine!
Thank you very much for your help!
mfg ar
Glad to hear its working! All the involved machines (real and virtualized) share the same ethernet segment so they need to have IPs from the same range. You can even assign IPs for the VMs via DHCP server, etc. Or to be more correct, you can have different network IP ranges, but then you need to route the traffic somehow... Radek
participants (3)
-
Andreas Rittershofer -
David Lutterkort -
Radek Hladik