11:13 a.m.
Abstraction of guest <--> host network connection in libvirt
=====================================
The <interface> element of a guest's domain config in libvirt has a
<source> element that describes what resources on a host will be used to
connect the guest's network interface to the rest of the world. This is
very flexible, allowing several different types of connection (virtual
network, host bridge, direct macvtap connection to physical interface,
qemu usermode, user-defined via an external script), but currently has
the problem that unnecessary details of the host config are embedded
into the guest's config; if the guest is migrated to a different host,
and that host has a different hardware or network config (or possibly
the same hardware, but that hardware is currently in use by a different
guest), the migration will fail.
I am proposing a change to libvirt's network XML that will allow us to
(optionally - old configs will remain valid) remove the host details
from the guest's domain XML (which can move around from host to host)
and place them in the network XML (which remains with the host); the
domain XML will then use existing config elements to associate each
guest interface with a "network".
The motivating use case for this change is the "direct" connection type
(which uses macvtap for vepa and vnlink connections directly between a
guest and a physical interface, rather than through a bridge), but it is
applicable for all types of connection. (Another hopeful side effect of
this change will be to make libvirt's network connection model easier to
realize on non-Linux hypervisors (eg, VMWare ESX), so Mathias - please
chime in!)
Background
--------------------
libvirt currently has 3 major types of guest interface connection (there
are also "type='user'" and "type='ethernet'", but they
probably wouldn't
be used in a multi-host environment, so I'm not considering them here):
1) type='network'
The guest's network interface is connected to a libvirt-created "virtual
network", which is in reality (in the case of KVM or Xen) a Linux bridge
device that isn't connected to any physical host interface - any
connection to the outside goes through the host's IP routing stack.
The network to use is indicated in the <source> element of the guest's
interface xml: <source network='mynetwork'/>. Because the name
'mynetwork' is controlled by libvirt, it's perfectly reasonable to
assume that the same network name could be available on another host
that is accepting a migrated guest.
2) type='bridge'
The guest's network interface is connected to a bridge device (eg "br0")
that has already been configured in the host's network config files (eg,
in /etc/sysconfig/network-scripts). This bridge is itself connected to
the outside via a physical host interface, eg "eth0", *NOT* through the
hosts IP routing stack.
The bridge to use is indicated in the source with <source
bridge='br0'/>. Although the naming of the bridge is outside the scope
of libvirt, it is at least possible to setup all hosts to have the same
bridge name (so that a guest could be migrated from one host to another).
3) type='direct'
The guest's network interface is connected directly to a physical
interface (eg "eth0") with macvtap, or sometimes to a virtual function
("VF") of a physical interface (which is also really just another
interface, from the software point of view).
The interface to use is indicated with <source interface='eth0'
mode='something'/> In this case, the interface name is determined by the
host OS and cannot be arbitrarily changed. Also a host will have
multiple interfaces / VFs available to guests, and in some modes may
allow only a single guest to connect to a given interface (implying that
the interface used by a guest when on one host will probably not be
available when migrating to another). So in order to have flexible
migration from one host to another, an abstraction to allow the guest
XML to use the same name on all hosts must be introduced.
Three possible methods for providing this abstraction come to mind:
Option 1
-----------
(Be forewarned that Option 1 & 2 are shown here mainly to illustrate my
thought process while arriving at my preferred Option - 3 :-)
In a manner similar to the way the vnet%d tap devices are created, name
the interface with an embedded variable (eg "eth%d") (plus attributes
for min and max %d) and let the underlying code in libvirt search
for/reserve an appropriate device>
This is the simplest to code/configure, but does not allow a) more
complex names (eg, interface names as determined by biosdevname can be
of the form "pci%dp%d_%d"), b) multiple ranges, c) oversubscribing of
interfaces (it is possible, although sub-optimal, to connect multiple
guest interfaces to a single host interface with macvtap).
VERDICT: looks ugly, not flexible enough.
Option 2
-----------
create a new class of libvirt XML config to describe a pool of network
interfaces, and reference this pool in the guest interface element:
<interface type='interfacePool'>
<source pool='red-network'/>
...
</interface>
The problem with this is that it requires a new API for
defining/undefining/etc management of "interface pools". Also, it
wouldn't allow (for example) one host to use a pool of macvtap addresses
to connect guests, and another host to use a host bridge for the same
connection (obviously, such a non-uniform setup wouldn't be desirable in
a large host farm, but may be encountered in some smaller setup)
VERDICT: creates more API clutter (ie extra work *and* confusion for
users). Is "flexible enough" for current motivation, but unnecessarily
limiting, eg doesn't help the model to be more easily adapted to VMWare etc.
Option 3
-----------
Up to now we've only discussed the need for separating the host-specific
config (<source> element) in the case of type='direct' interfaces (well,
in reality I've gone back and edited this document so many times that is
no longer true, but play along with me! :-). But it really is a problem
for all interface types - all of the information currently in the
guest's interface <source> element really is tied to the host, and
shouldn't be defined in detail in the guest XML; it should instead be
defined once for each host, and only referenced by some name in the
guest XML; that way as a guest moves from host to host, it will
automatically adjust its connection to match the new environmant.
As a more general solution, instead of having the special new
"interfacePool" object in the config, what if the XML for "network was
expanded to mean "any type of guest network connection" (with a new
"type='xxx'" attribute at the toplevel to indicate which type), not just
"a private bridge optionally connected to the real world via routing/NAT"?
If this was the case, the guest interface XML could always be, eg:
<interface type='network'>
<source network='red-network'/>
...
</interface>
and depending on the network config of the host the guest was migrated
to, this could be either a direct (macvtap) connection via an interface
allocated from a pool (the pool being defined in the definition of
'red-network'), a bridge (again, pointed to by the definition of
'red-network', or a virtual network (using the current network
definition syntax). This way the same guest could be migrated not only
between macvtap-enabled hosts, but from there to a host using a bridge,
or maybe a host in a remote location that used a virtual network with a
secure tunnel to connect back to the rest of the red-network. (Part of
the migration process would of course check that the destination host
had a network of the proper name, and fail if it didn't; management
software at a level above libvirt would probably filter a list of
candidate migration destinations based on available networks, and only
attempt migration to one that had the matching network available).
Examples of 'red-network' for different types of connections (all of
these would work with the interface XML given above):
<!-- Existing usage - a libvirt virtual network -->
<network> <!-- (you could put "type='virtual'" here for
symmetry) -->
<name>red-network</name>
<bridge name='virbr0'/>
<forward mode='route'/>
...
</network>
<!-- The simplest - an existing host bridge -->
<network type='bridge'>
<name>red-network</name>
<bridge name='br0'/>
</network>
<network type='direct'>
<name>red-network</name>
<source mode='vepa'>
<!-- define the pool of available interfaces here. Interfaces may have -->
<!-- parameters associated with them, eg max number of simultaneous
guests -->
</source>
<!-- add any other elements from the guest interface XML that are tied
to -->
<!-- the host here (virtualport ?) (of course if they're host specific,
they -->
<!-- should have been in <source> in the first place!!) -->
</network>
I know there may be some resistance to this expansion of the usage of
<network>, but I think it does fit in with the current usage properly,
and is preferable to adding an entire new class of API just to define a
pool of interfaces.
Open questions:
1) What should the <pool> element inside network/source look like.
Making each interface in the pool a separate element, with possible
attributed, would be the simplest to code, but would get tedious on a
system with, for example, an ethernet card with 64 VFs. On the other
hand, just parameterizing a string (eth%d) is inadequate, eg, when there
are multiple non-contiguous ranges.
2) do we need a "max connections" for each interface in a pool of
macvtap interfaces? Or should we just overload them in a round-robin
fashion unless mode='passthru' (a new mode which requires only one guest
per interface).
3) What about the parameters in the <virtualport> element that are
currently used by vepa/vnlink. Do those belong with the host, or with
the guest?
4) Are there other <network> types that we want? Perhaps the recent
proposal for IPSec / secure tunnels could be incorporated as a new
network type (or maybe it could just be the standard "virtual" type,
with a tunnel as the forward device).