5:34 p.m.
Using a bridge to connect a qemu NIC to a host interface offers a fair
amount of flexibility to reconfigure the host without restarting the VM.
For example, if the bridge connects host interface eth0 to the qemu tap0
interface, eth0 can be hot-removed and hot-plugged without affecting the
VM. Similarly, if the bridge connects host VLAN interface vlan0 to the
qemu tap0 interface, the admin can easily replace vlan0 with vlan1
without the VM noticing.
Using the macvtap driver instead of a kernel bridge, the host interface
is much more tightly tied to the VM. Qemu communicates with the macvtap
interface through a file descriptor, and the macvtap interface is bound
permanently to a specific host interface when it is first created.
What's more, if the underlying host interface disappears, the macvtap
interface vanishes along with it, leaving the VM holding a file
descriptor for a deleted file.
To avoid race conditions during system startup, I would like libvirt to
allow starting up the VM with a NIC even if the underlying host
interface doesn't yet exist, deferring creation of the macvtap interface
(analogous to starting up the VM with a tap interface bound to an orphan
bridge). To support adding and removing a host interface without
restarting the VM, I would like libvirt to react to the (re)appearance
of the underlying host interface, creating a new macvtap interface and
passing the new fd to qemu to reconnect to the NIC.
(It would also be nice if libvirt allowed the user to change which
underlying host interface the qemu NIC is connected to. I'm ignoring
this issue for now, except to note that implementing the above features
should make this easier.)
The libvirt API already supports domainAttachDevice and
domainDetachDevice to add or remove an interface while the VM is
running. In the qemu implementation, these commands add or remove the
VM NIC device as well as reconfiguring the host side. This works only
if the OS and application running in the VM can handle PCI hotplug and
dynamically reconfigure its network. I would like to isolate the VM
from changes to the host network setup, whether you use macvtap or a
bridge.
The changes I think are needed to implement this include:
1. Refactor qemudDomainAttachNetDevice/qemudDomainDetachNetDevice, which
currently handle both backend (host) setup and adding/removing the VM
NIC device; move the backend setup code into separate functions that can
called separately without affecting VM devices.
2. Implement a thread or task that watches for changes to the underlying
host interface for each configured macvtap interface, and reacts by
invoking the appropriate backend setup code.
3. Change qemudBuildCommandLine to defer backend setup if qemu supports
the necessary features for doing it later (e.g. the host_net_add monitor
command).
4. Implement appropriate error handling and reporting, and any necessary
changes to the configuration schema.
The following patches are a partial implementation of the above as a
proof of concept.
Patch 1 implements change (1) above, moving the backend setup code to
new functions
qemudDomainConnectNetBackend/qemudDomainDisconnectNetBackend, and
calling these functions from the existing
qemudDomainAttachNetDevice/qemudDomainDetachNetDevice. I think this
change is useful on its own: it breaks up two monster functions into
more manageable pieces, and eliminates some code duplication (e.g. the
try_remove clause at the end of qemudDomainAttachNetDevice).
Patch 2 is a godawful hack roughly implementing change (2) above (did I
mention that this is a proof of concept?). It spawns a thread that
simply tries reconnecting the backend of each macvtap interface once a
second. As long as the interface is already up, the reconnection fails.
If the macvtap interface goes away because the underlying host interface
disappears, the reconnection fails until the host interface reappears.
I ran into two major issues while implementing change (2):
- Can we use the existing virEvent functions to invoke the reconnection
process, triggered either by a timer or by an event from the host? It
seems like this ought to work, but it appears that communication between
libvirt and the qemu monitor relies on an event, and since all events
run in the same thread, there's no way for an event to call the monitor.
- Should the reconnection process use udev or hal to get notifications,
or leverage the node device code which itself uses udev or hal?
Currently there doesn't appear to be a way to get notifications of
changes to node devices; if there were, we'd still need to address the
threading issue. If we use node devices, what changes to the
configuration schema would be needed to associate a macvtap interface
with the underlying node device?
I didn't attempt to implement changes (3) or (4) above but would welcome
input on those items as well.
--Ed