On 29.06.2012 08:09, Eiichi Tsukata wrote: > Hello, > > This is a patch to fix vm's outbound traffic control problem. > > Currently, vm's outbound traffic control by libvirt doesn't go well. > This problem was previously discussed at libvir-list ML, however > it seems that there isn't still any answer to the problem. > http://www.redhat.com/archives/libvir-list/2011-August/msg00333.html > > I measured Guest(with virtio-net) to Host TCP throughput with the > command "netperf -H". > Here are the outbound QoS parameters and the results. > > outbound average rate[kilobytes/s] : Guest to Host throughput[Mbit/s] > ====================================================================== > 1024 (8Mbit/s) : 4.56 > 2048 (16Mbit/s) : 3.29 > 4096 (32Mbit/s) : 3.35 > 8192 (64Mbit/s) : 3.95 > 16384 (128Mbit/s) : 4.08 > 32768 (256Mbit/s) : 3.94 > 65536 (512Mbit/s) : 3.23 > > The outbound traffic goes down unreasonably and is even not controled. > > The cause of this problem is too large mtu value in "tc filter" command run by > libvirt. The command uses burst value to set mtu and the burst is equal to > average rate value if it's not set. This value is too large. For example > if the average rate is set to 1024 kilobytes/s, the mtu value is set to 1024 > kilobytes. That's too large compared to the size of network packets. > Here libvirt applies tc ingress filter to Host's vnet(tun) device. > Tc ingress filter is implemented with TBF(Token Buckets Filter) algorithm. TBF > uses mtu value to calculate the amount of token consumed by each packet. With too > large mtu value, the token consumption rate is set too large. This leads to > token starvation and deterioration of TCP throughput. > > Then, should we use the default mtu value 2 kilobytes? > The anser is No, because Guest with virtio-net device uses 65536 bytes > as mtu to transmit packets to Host, and the tc filter with the default mtu > value 2k drops packets whose size is larger than 2k. So, the most packets > is droped and again leads to deterioration of TCP throughput. > > The appropriate mtu value is 65536 bytes which is equal to the maximum value > of network interface device defined in <linux/netdevice.h>. The value is > not so large that it causes token starvation and not so small that it > drops most packets. > Therefore this patch set the mtu value to 64kb(== 65535 bytes). > > Again, here are the outbound QoS parameters and the TCP throughput with > the libvirt patched. > > outbound average rate[kilobytes/s] : Guest to Host throughput[Mbit/s] > ====================================================================== > 1024 (8Mbit/s) : 8.22 > 2048 (16Mbit/s) : 16.42 > 4096 (32Mbit/s) : 32.93 > 8192 (64Mbit/s) : 66.85 > 16384 (128Mbit/s) : 133.88 > 32768 (256Mbit/s) : 271.01 > 65536 (512Mbit/s) : 547.32 > > The outbound traffic conforms to the given limit. > > Thank you, > > Signed-off-by: Eiichi Tsukata <eiichi.tsukata.xh(a)hitachi.com&gt; > --- > AUTHORS | 1 + > src/util/virnetdevbandwidth.c | 2 +- > 2 files changed, 2 insertions(+), 1 deletions(-) > > diff --git a/AUTHORS b/AUTHORS > index b876ee6..375db24 100644 > --- a/AUTHORS > +++ b/AUTHORS > @@ -246,6 +246,7 @@ Patches have also been contributed by: > Gerd Hoffmann <kraxel(a)redhat.com&gt; > Viktor Mihajlovski <mihajlov(a)linux.vnet.ibm.com&gt; > Thang Pham <thang.pham(a)us.ibm.com&gt; > + Eiichi Tsukata <eiichi.tsukata.xh(a)hitachi.com&gt; > > [....send patches to get your name here....] > > diff --git a/src/util/virnetdevbandwidth.c b/src/util/virnetdevbandwidth.c > index b9bd2e3..93df5c1 100644 > --- a/src/util/virnetdevbandwidth.c > +++ b/src/util/virnetdevbandwidth.c > @@ -136,7 +136,7 @@ virNetDevBandwidthSet(const char *ifname, > virCommandAddArgList(cmd, "filter", "add", "dev", ifname, "parent", > "ffff:", "protocol", "ip", "u32", "match", "ip", > "src", "0.0.0.0/0", "police", "rate", average, > - "burst", burst, "mtu", burst, "drop", "flowid", > + "burst", burst, "mtu", "64kb", "drop", "flowid", > ":1", NULL); > > if (virCommandRun(cmd, NULL) < 0) > Yeah, this tc magic was the hardest part when I've introduced QoS feature. I tested it on small values where this has not fully showed. ACK and pushed. Thanks!