Re: [libvirt] [RFC] [Patch] Support for Linux macvtap device

On Mon, Jan 25, 2010 at 12:47:17PM -0500, Stefan Berger wrote: I don't think this is the correct place to be adding this kind of configuration / functionality. The virNetworkPtr / <network> XML is describing a virtual network capability which is *not* directly connected to the LAN. It may be configured to route from the virtual network to the LAN, with optional NAT applied. So while the implementation may use a bridge device, this bridge is not connected to any physical device. Since VEPA is about directly connecting VMs to the LAN, this doesn't really fit here. I'm not sure how familiar people are with the current libvirt object model, so here's a 30 second overview - virDomainPtr / <domain> - management & configuration of the virtual machines & their hardware. - virNetworkPtr / <network> - virtual networks, either isolated or routed (+NAT) to the LAN - virInterfacePtr / <interface> - host interface configuration for physical devices, bridge, bonding & vlan devices - virNodeDevPtr / <device> - host device enumeration / discovery One warning.. - The <domain> XML also contains a element called <interface> which declares a guest NIC. Be careful not to mix up with the similarly named virInterfacePtr XML also using <interface> refering to a host NIC. In the context of bridging a guest to a plain ethernet device, these fit together as follows 1. The virNodeDevPtr APIs are used to discover what physical network devices exist, 'eth0' 2. The virInterfacePtr APIs are used to create a bridge on the host br0, containing the physical device 'eth0' 3. The virDomainPtr APIs are used to define a guest configuration using guest NIC <interface type='bridge'> element, whcih uses a plain TAP device. The logical setup of devices is /-> tap0 -> vm0 pcidev0 -> eth0 -> br0 --> tap1 -> vm1 \-> tap2 -> vm2 Now, throw SR-IOV into the picture.... With SR-IOV there is a single PCI card, with multiple virtual functions. The number of virtual functions is fixed at time of kernel module load so there's no live configuration available from userspace. Every virtual function shows up as a new PCI device in the virNodeDevPtr APIs, and each of those has an associated ethX device visible too. So with SR-IOV plus traditional bridging the same 3 step process above more or less applies unchanged, creating a bridge for each SR-IOV virtual function ethX device & then attaching guests using TAP. The logical setup of devices is a little different pcidev0 -> eth0 -> br0 -> tap0 -> vm0 | pcidev1 -> eth1 -> br1 -> tap1 -> vm1 | pcidev2 -> eth2 -> br2 -> tap2 -> vm2 and you can correlate pcidev0/1/2 to determine they are all virtual function. Of course there's no reason this has to restrict itself to a 1 bridge == 1 guest arrangement - i've just kept this diagram simple. So every guest VM NIC gets a new uniqely named macvtap device, and and unqiue TAP device ? IIUC, with a macvlan + a regular NIC we'd thus be seeing something like this arrangement .. /-> macvtap0 -> tap0 -> vm0 pcidev0 -> eth0 --> macvtap1 -> tap1 -> vm1 \-> macvtap2 -> tap2 -> vm2 I guess you could also use this with an SR-IOV card too pcidev0 -> eth0 -> macvtap0 -> tap0 -> vm0 | pcidev1 -> eth1 -> macvtap1 -> tap1 -> vm1 | pcidev2 -> eth2 -> macvtap2 -> tap2 -> vm2 Also, again there's no reason why we'd need to restrict ourselves to only using a single macvtap device with each ethX device / virtual function. The core question to me is whether we need to introduce any new configuration parameters at the host level. It looks to me like it can all be done against the guest XML <interface> element. Specifically, I don't see a need to manage the macvtapN devices directly via the virInterfacePtr APIs. This is good, because that is all delegated to the netcf library, which in turn is just a way to read/write the /etc/sysconfig/networking-scripts. These ifcfg-XXX files have no knowledge of macvtap, and so plumbing all that through the networking-scripts, netcf and libvirt virInterfacePtr APIs is non-trivial work we'd be better off avoiding unless there was a clear compelling need. So unless I'm missing something major in my reasoning here I think in the domain XML we end up with two possible configs for guest network interfaces 1. The current one using plain Linux software bridging, which we can't change in an incompatible way <interface type='bridge'/> <source bridge='br0'/> <target dev='vnet0'/> </interface> Here, the source device is a bridge previously setup to have a physical device enslaved (regular or SR-IOV) The target device is the plain TAP device 2. A new one using hardware bridging, which we can freely define for our new needs <interface type='direct'/> <source dev='eth0' mode='vepa|pepa|bridge'/> <target dev='vnet0'/> </interface> Here, source device is a physical device (regular or SR-IOV). The target device is a macvtap device. In both cases the TAP or macvtap device is created on the fly when the VM is booted & destroyed at shutdown (either by the kernel, or manually by libvirt for macvtap). Both configs are ultimately a form of 'bridging', the difference is that the former is focused on software bridges, as explicitly managed objects separate from the physical device, while the latter is hardware bridges which don't need to be managed separately. Ok, this plan all makes sense. The only thing is that if we want to allow the user to manually specify a macvtap name, we should declare that they are not allowed to use the prefix 'macvtap' for their name. We do similar with TAP, where we declare 'vnet' is a reserved prefix for auto-generated names. As mentioned earlier, we don't want to touch the VIR_DOMAIN_NET_TYPE_NETWORK case, since that's not bridging - its separate virtal networks using NAT. If my idea above is acceptable, we'd be adding a new VIR_DOMAIN_NET_TYPE_DIRECT that we'd hook up. As an interface for internal usage, this looks like a fine start. [snip]. I've not had time to look at the details of this macvtap.c code yet, but I assume its doing all you need :-) Is there any benefit to using the network libnl.so library, rather than the ioctl()'s directly ? I'm not too familiar with either, but we already use libnl.so indirectly via netcf, so don't be afraid of adding a dependancy on the libnl.so library if you consider it to be useful. This file shouldn't get any changes really. Also don't think we need to be adding extra public APIs for this, if we do it within scope of the domain XML. For the next submission, it'd be very helpful for review if you could split it into 3 patches 1. Adding the extra domain XML syntax to src/conf/domain_conf.{h,c} and docs/schemas/domain.rng 2. Adding the macvtap helper code in src/util/macvtap.{h,c} 3. Implementation for the QEMU driver to use the stuff from the first two patches. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|