[libvirt] RFC: disconnecting guest/domain interface config from host config (aka migration with macvtap)
Abstraction of guest <--> host network connection in libvirt ===================================== The <interface> element of a guest's domain config in libvirt has a <source> element that describes what resources on a host will be used to connect the guest's network interface to the rest of the world. This is very flexible, allowing several different types of connection (virtual network, host bridge, direct macvtap connection to physical interface, qemu usermode, user-defined via an external script), but currently has the problem that unnecessary details of the host config are embedded into the guest's config; if the guest is migrated to a different host, and that host has a different hardware or network config (or possibly the same hardware, but that hardware is currently in use by a different guest), the migration will fail. I am proposing a change to libvirt's network XML that will allow us to (optionally - old configs will remain valid) remove the host details from the guest's domain XML (which can move around from host to host) and place them in the network XML (which remains with the host); the domain XML will then use existing config elements to associate each guest interface with a "network". The motivating use case for this change is the "direct" connection type (which uses macvtap for vepa and vnlink connections directly between a guest and a physical interface, rather than through a bridge), but it is applicable for all types of connection. (Another hopeful side effect of this change will be to make libvirt's network connection model easier to realize on non-Linux hypervisors (eg, VMWare ESX), so Mathias - please chime in!) Background -------------------- libvirt currently has 3 major types of guest interface connection (there are also "type='user'" and "type='ethernet'", but they probably wouldn't be used in a multi-host environment, so I'm not considering them here): 1) type='network' The guest's network interface is connected to a libvirt-created "virtual network", which is in reality (in the case of KVM or Xen) a Linux bridge device that isn't connected to any physical host interface - any connection to the outside goes through the host's IP routing stack. The network to use is indicated in the <source> element of the guest's interface xml: <source network='mynetwork'/>. Because the name 'mynetwork' is controlled by libvirt, it's perfectly reasonable to assume that the same network name could be available on another host that is accepting a migrated guest. 2) type='bridge' The guest's network interface is connected to a bridge device (eg "br0") that has already been configured in the host's network config files (eg, in /etc/sysconfig/network-scripts). This bridge is itself connected to the outside via a physical host interface, eg "eth0", *NOT* through the hosts IP routing stack. The bridge to use is indicated in the source with <source bridge='br0'/>. Although the naming of the bridge is outside the scope of libvirt, it is at least possible to setup all hosts to have the same bridge name (so that a guest could be migrated from one host to another). 3) type='direct' The guest's network interface is connected directly to a physical interface (eg "eth0") with macvtap, or sometimes to a virtual function ("VF") of a physical interface (which is also really just another interface, from the software point of view). The interface to use is indicated with <source interface='eth0' mode='something'/> In this case, the interface name is determined by the host OS and cannot be arbitrarily changed. Also a host will have multiple interfaces / VFs available to guests, and in some modes may allow only a single guest to connect to a given interface (implying that the interface used by a guest when on one host will probably not be available when migrating to another). So in order to have flexible migration from one host to another, an abstraction to allow the guest XML to use the same name on all hosts must be introduced. Three possible methods for providing this abstraction come to mind: Option 1 ----------- (Be forewarned that Option 1 & 2 are shown here mainly to illustrate my thought process while arriving at my preferred Option - 3 :-) In a manner similar to the way the vnet%d tap devices are created, name the interface with an embedded variable (eg "eth%d") (plus attributes for min and max %d) and let the underlying code in libvirt search for/reserve an appropriate device> This is the simplest to code/configure, but does not allow a) more complex names (eg, interface names as determined by biosdevname can be of the form "pci%dp%d_%d"), b) multiple ranges, c) oversubscribing of interfaces (it is possible, although sub-optimal, to connect multiple guest interfaces to a single host interface with macvtap). VERDICT: looks ugly, not flexible enough. Option 2 ----------- create a new class of libvirt XML config to describe a pool of network interfaces, and reference this pool in the guest interface element: <interface type='interfacePool'> <source pool='red-network'/> ... </interface> The problem with this is that it requires a new API for defining/undefining/etc management of "interface pools". Also, it wouldn't allow (for example) one host to use a pool of macvtap addresses to connect guests, and another host to use a host bridge for the same connection (obviously, such a non-uniform setup wouldn't be desirable in a large host farm, but may be encountered in some smaller setup) VERDICT: creates more API clutter (ie extra work *and* confusion for users). Is "flexible enough" for current motivation, but unnecessarily limiting, eg doesn't help the model to be more easily adapted to VMWare etc. Option 3 ----------- Up to now we've only discussed the need for separating the host-specific config (<source> element) in the case of type='direct' interfaces (well, in reality I've gone back and edited this document so many times that is no longer true, but play along with me! :-). But it really is a problem for all interface types - all of the information currently in the guest's interface <source> element really is tied to the host, and shouldn't be defined in detail in the guest XML; it should instead be defined once for each host, and only referenced by some name in the guest XML; that way as a guest moves from host to host, it will automatically adjust its connection to match the new environmant. As a more general solution, instead of having the special new "interfacePool" object in the config, what if the XML for "network was expanded to mean "any type of guest network connection" (with a new "type='xxx'" attribute at the toplevel to indicate which type), not just "a private bridge optionally connected to the real world via routing/NAT"? If this was the case, the guest interface XML could always be, eg: <interface type='network'> <source network='red-network'/> ... </interface> and depending on the network config of the host the guest was migrated to, this could be either a direct (macvtap) connection via an interface allocated from a pool (the pool being defined in the definition of 'red-network'), a bridge (again, pointed to by the definition of 'red-network', or a virtual network (using the current network definition syntax). This way the same guest could be migrated not only between macvtap-enabled hosts, but from there to a host using a bridge, or maybe a host in a remote location that used a virtual network with a secure tunnel to connect back to the rest of the red-network. (Part of the migration process would of course check that the destination host had a network of the proper name, and fail if it didn't; management software at a level above libvirt would probably filter a list of candidate migration destinations based on available networks, and only attempt migration to one that had the matching network available). Examples of 'red-network' for different types of connections (all of these would work with the interface XML given above): <!-- Existing usage - a libvirt virtual network --> <network> <!-- (you could put "type='virtual'" here for symmetry) --> <name>red-network</name> <bridge name='virbr0'/> <forward mode='route'/> ... </network> <!-- The simplest - an existing host bridge --> <network type='bridge'> <name>red-network</name> <bridge name='br0'/> </network> <network type='direct'> <name>red-network</name> <source mode='vepa'> <!-- define the pool of available interfaces here. Interfaces may have --> <!-- parameters associated with them, eg max number of simultaneous guests --> </source> <!-- add any other elements from the guest interface XML that are tied to --> <!-- the host here (virtualport ?) (of course if they're host specific, they --> <!-- should have been in <source> in the first place!!) --> </network> I know there may be some resistance to this expansion of the usage of <network>, but I think it does fit in with the current usage properly, and is preferable to adding an entire new class of API just to define a pool of interfaces. Open questions: 1) What should the <pool> element inside network/source look like. Making each interface in the pool a separate element, with possible attributed, would be the simplest to code, but would get tedious on a system with, for example, an ethernet card with 64 VFs. On the other hand, just parameterizing a string (eth%d) is inadequate, eg, when there are multiple non-contiguous ranges. 2) do we need a "max connections" for each interface in a pool of macvtap interfaces? Or should we just overload them in a round-robin fashion unless mode='passthru' (a new mode which requires only one guest per interface). 3) What about the parameters in the <virtualport> element that are currently used by vepa/vnlink. Do those belong with the host, or with the guest? 4) Are there other <network> types that we want? Perhaps the recent proposal for IPSec / secure tunnels could be incorporated as a new network type (or maybe it could just be the standard "virtual" type, with a tunnel as the forward device).
On 04/12/2011 12:13 PM, Laine Stump wrote:
Abstraction of guest <--> host network connection in libvirt =====================================
I notice that this RFE doesn't mention openvswitch at all. Wouldn't
Here is a response that was posted in the tracking bug report (https://bugzilla.redhat.com/show_bug.cgi?id=643947). I'm transferring it to here and replying to keep the discussion in one place: Martin Wilck writes: that offer
all you need, and even more, if there was a plugin for configuring openvswitch in libvirt?
Perhaps having "vSwitch" in the title of this bug is confusing things... openvswitch is an alternative to using a 1) linux host bridge, 2) a libvirt virtual network (which is also a host bridge in disguise) or 3) a direct connection to an interface. It is *not*, as far as I understand, something that could/should be used together with any of those, but *instead of* them. There are libvirt users who are using macvtap for direct connection between guests and the host network interface device 1) for performance reasons, and 2) because libvirt's macvtap support also has support for physical switches that have vepa and vnlink - in these modes all traffic to/from a guest is mandated to travel through the physical switch, even if it ends up hairpinning back to the same host. This allows the admin of the physical switch to enforce rules about type of traffic, QoS, etc. openvswitch would not be interesting in this scenario, because it adds extra overhead on the host, and also allows bypassing the mandatory trip to the physical switch. The main purpose of the proposal here is to allow guests using this type of direct connection to migrate successfully among hosts. A nice side effect is that it will make it easier to add in support for things like openvsitch (which, as I understand it, can just become another type of libvirt <network>). So while openvswitch may benefit from this code (see below), it is not a viable alternative to it.
Option 3 -----------
Up to now we've only discussed the need for separating the host-specific config (<source> element) in the case of type='direct' interfaces (well, in reality I've gone back and edited this document so many times that is no longer true, but play along with me! :-). But it really is a problem for all interface types - all of the information currently in the guest's interface <source> element really is tied to the host, and shouldn't be defined in detail in the guest XML; it should instead be defined once for each host, and only referenced by some name in the guest XML; that way as a guest moves from host to host, it will automatically adjust its connection to match the new environmant.
As a more general solution, instead of having the special new "interfacePool" object in the config, what if the XML for "network was expanded to mean "any type of guest network connection" (with a new "type='xxx'" attribute at the toplevel to indicate which type), not just "a private bridge optionally connected to the real world via routing/NAT"?
If this was the case, the guest interface XML could always be, eg:
<interface type='network'> <source network='red-network'/> ... </interface>
and depending on the network config of the host the guest was migrated to, this could be either a direct (macvtap) connection via an interface allocated from a pool (the pool being defined in the definition of 'red-network'), a bridge (again, pointed to by the definition of 'red-network', or a virtual network (using the current network definition syntax). This way the same guest could be migrated not only between macvtap-enabled hosts, but from there to a host using a bridge, or maybe a host in a remote location that used a virtual network with a secure tunnel to connect back to the rest of the red-network. (Part of the migration process would of course check that the destination host had a network of the proper name, and fail if it didn't; management software at a level above libvirt would probably filter a list of candidate migration destinations based on available networks, and only attempt migration to one that had the matching network available).
Examples of 'red-network' for different types of connections (all of these would work with the interface XML given above):
<!-- Existing usage - a libvirt virtual network --> <network> <!-- (you could put "type='virtual'" here for symmetry) --> <name>red-network</name> <bridge name='virbr0'/> <forward mode='route'/> ... </network>
<!-- The simplest - an existing host bridge --> <network type='bridge'> <name>red-network</name> <bridge name='br0'/> </network>
<network type='direct'> <name>red-network</name> <source mode='vepa'> <!-- define the pool of available interfaces here. Interfaces may have --> <!-- parameters associated with them, eg max number of simultaneous guests --> </source> <!-- add any other elements from the guest interface XML that are tied to --> <!-- the host here (virtualport ?) (of course if they're host specific, they --> <!-- should have been in <source> in the first place!!) --> </network>
In other words, to support openvswitch, we would add: <network type='openvswitch'> <name>red-network</name> <!-- whatever XML is necessary to configure the openvswitch --> </network> Then at guest boot startup time, libvirt would do all the setup to connect the qemu process' network interface to the openvswitch.
On 04/27/2011 09:58 AM, Oved Ourfalli wrote:
Laine, hello.
We read your proposal for abstraction of guest<--> host network connection in libvirt.
You has an open issue there regarding the vepa/vnlink attributes: "3) What about the parameters in the<virtualport> element that are currently used by vepa/vnlink. Do those belong with the host, or with the guest?"
The parameters for the virtualport element should be on the guest, and not the host, because a specific interface can run multiple profiles,
Are you talking about host interface or guest interface? If you mean that multiple different profiles can be used when connecting to a particular switch - as long as there are only a few different profiles, rather than each guest having its own unique profile, then it still seems better to have the port profile live with the network definition (and just define multiple networks, one for each port profile).
so it will be a mistake to define a profile to be interface specific on the host. Moreover, putting it in the guest level will enable us in the future (if supported by libvirt/qemu) to migrate a vm from a host with vepa/vnlink interfaces, to another host with a bridge, for example.
It seems to me like doing exactly the opposite would make it easier to migrate to a host that used a different kind of switching (from vepa to vnlink, or from a bridged interface to vepa, etc), since the port profile required for a particular host's network would be at the host waiting to be used.
So, in the networks at the host level you will have: <network type='direct'> <name>red-network</name> <source mode='vepa'> <pool> <interface> <name>eth0</name> ..... </interface> <interface> <name>eth4</name> ..... </interface> <interface> <name>eth18</name> ..... </interface> </pool> </source> </network>
And in the guest you will have (for vepa): <interface type='network'> <source network='red-network'/> <virtualport type="802.1Qbg"> <parameters managerid="11" typeid="1193047" typeidversion="2" instanceid="09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f"/> </virtualport> </interface>
Or (for vnlink): <interface type='network'> <source network='red-network'/> <virtualport type="802.1Qbh"> <parameters profile_name="profile1"/> </virtualport> </interface>
This illustrates the problem I was wondering about - in your example it would not be possible for the guest to migrate from the host using a vepa switch to the host using a vnlink switch (and it would be possible to migrate to a host using a standard bridge only if the virtualport element was ignored). If the virtualport element lived with the network definition of red-network on each host, it could be migrated without problem. The only problematic thing would be if any of the attributes within <parameters> was unique for each guest (I don't know anything about the individual attributes, but "instanceid" sounds like it might be different for each guest).
Then, when migrating from a vepa/vnlink host to another vepa/vnlink host containing red-network, the profile attributes will be available at the guest domain xml. In case the target host has a red-network, which isn't vepa/vnlink, we want to be able to choose whether to make the use of the profile attributes optional (i.e., libvirt won't fail in case of migrating to a network of another type), or mandatory (i.e., libvirt will fail in case of migration to a non-vepa/vnlink network).
We have something similar in CPU flags: <cpu match="exact"> <model>qemu64</model> <topology sockets="S" cores="C" threads="T"/> <feature policy="require/optional/disable......" name="sse2"/> </cpu>
In this analogy, does "CPU flags" == "mode (vepa/vnlink/bridge)" or does "CPU flags" == "virtualport parameters" ? It seems like what you're wanting can be satisfied by simply not defining "red-network" on the hosts that don't have the proper networking setup available (maybe what you *really* want to call it is "red-vnlink-network").
participants (1)
-
Laine Stump