[libvirt] [RFC PATCH] set tap mtu

Below is a simple PoC for setting a large MTU size on a tap device. With this we are able to improve net i/o throughput substantially (~40% improvement on TX and ~130% improvement on RX). This is just RFC because it's hardcoded to an MTU of 9000 for any tap device. Thoughts on the best way to add this kind of support? diff -up libvirt-0.4.6/src/bridge.c~tap-mtu libvirt-0.4.6/src/bridge.c --- libvirt-0.4.6/src/bridge.c~tap-mtu 2008-08-29 05:19:52.000000000 -0700 +++ libvirt-0.4.6/src/bridge.c 2008-11-17 18:37:56.000000000 -0800 @@ -276,6 +276,38 @@ brDeleteInterface(brControl *ctl ATTRIBU #endif /** + * ifSetMtu: + * @ctl: bridge control pointer + * @ifname: interface name to set MTU for + * @mtu: MTU value + * + * This function sets the @mtu for a given interface @ifname. Typically + * used on a tap device to set up for Jumbo Frames. + * + * Returns 0 in case of success or an errno code in case of failure. + */ +static int +ifSetMtu(brControl *ctl, const char *ifname, int mtu) +{ + struct ifreq ifr; + int len; + + if (!ctl || !ifname) + return EINVAL; + + if ((len = strlen(ifname)) >= BR_IFNAME_MAXLEN) + return EINVAL; + + memset(&ifr, 0, sizeof(struct ifreq)); + + strncpy(ifr.ifr_name, ifname, len); + ifr.ifr_name[len] = '\0'; + ifr.ifr_mtu = mtu; + + return ioctl(ctl->fd, SIOCSIFMTU, &ifr) == 0 ? 0 : errno; +} + +/** * brAddTap: * @ctl: bridge control pointer * @bridge: the bridge name @@ -334,6 +366,8 @@ brAddTap(brControl *ctl, } if (ioctl(fd, TUNSETIFF, &try) == 0) { + if ((errno = ifSetMtu(ctl, try.ifr_name, 9000))) + goto error; if ((errno = brAddInterface(ctl, bridge, try.ifr_name))) goto error; if ((errno = brSetInterfaceUp(ctl, try.ifr_name, 1)))

On Mon, Nov 17, 2008 at 06:39:45PM -0800, Chris Wright wrote:
Below is a simple PoC for setting a large MTU size on a tap device. With this we are able to improve net i/o throughput substantially (~40% improvement on TX and ~130% improvement on RX). This is just RFC because it's hardcoded to an MTU of 9000 for any tap device. Thoughts on the best way to add this kind of support?
Well if we want it to be configurable per guest then we'd add it to the XML, in the <target> device field, alongside the VIF name, eg <interface type='network'> <source network='default'/> <target dev='vnet7' mtu='9000'/> <mac address="11:22:33:44:55:66"/> </interface> libvirt can handle this stuff directly for QEMU and LXC drivers but for Xen, VIF setup is done by an /etc/xen/scripts/vif-bridge script. AFAIK, XenD has no config param for MTU, but there's no reason we can't write a patch for XenD to accept an MTU param and pass that through to vif-bridge. If large MTUs are useful for Xen's netback of course... If we don't need this configurable, is there any downside to setting a MTU of 9000 for all TAP devices we create. I assume that PTMUD will ensure that the guest only sends packets <= 1500 if the physical NIC connected to the bridge doesn't have such a large MTU, or if the guest doesn't do PMTUD, then the bridge code will do fragementation as needed ? Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Tue, 2008-11-18 at 14:57 +0000, Daniel P. Berrange wrote:
On Mon, Nov 17, 2008 at 06:39:45PM -0800, Chris Wright wrote:
Below is a simple PoC for setting a large MTU size on a tap device. With this we are able to improve net i/o throughput substantially (~40% improvement on TX and ~130% improvement on RX). This is just RFC because it's hardcoded to an MTU of 9000 for any tap device. Thoughts on the best way to add this kind of support?
Well if we want it to be configurable per guest then we'd add it to the XML, in the <target> device field, alongside the VIF name, eg
<interface type='network'> <source network='default'/> <target dev='vnet7' mtu='9000'/> <mac address="11:22:33:44:55:66"/> </interface>
Yeah, I was just about to suggest that.
libvirt can handle this stuff directly for QEMU and LXC drivers but for Xen, VIF setup is done by an /etc/xen/scripts/vif-bridge script. AFAIK, XenD has no config param for MTU, but there's no reason we can't write a patch for XenD to accept an MTU param and pass that through to vif-bridge. If large MTUs are useful for Xen's netback of course...
If we don't need this configurable, is there any downside to setting a MTU of 9000 for all TAP devices we create. I assume that PTMUD will ensure that the guest only sends packets <= 1500 if the physical NIC connected to the bridge doesn't have such a large MTU, or if the guest doesn't do PMTUD, then the bridge code will do fragementation as needed ?
My take on it is that it needs to be an opt in thing - if path MTU discovery was sufficient to deal with all cases then we'd probably have an mtu > 1500 everywhere ... Herbert? Cheers, Mark.

On Tue, Nov 18, 2008 at 03:16:14PM +0000, Mark McLoughlin wrote:
My take on it is that it needs to be an opt in thing - if path MTU discovery was sufficient to deal with all cases then we'd probably have an mtu > 1500 everywhere ... Herbert?
PMTU often breaks due to firewalls. Also even if you have did a large MTU it is only effective if the entire path supports it. So for most peers you'll be using an MTU of 1500 or less. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
participants (4)
-
Chris Wright
-
Daniel P. Berrange
-
Herbert Xu
-
Mark McLoughlin