On 08/10/2011 02:55 AM, Upendra Moturi wrote: Hi, does this patch solve your problem? I am setting the MTU to fixed 2kb. Doing tests with scp seems to indicate that this improved the situation -- at least for me. Stefan Signed-off-by: Stefan Berger <stefanb(a)linux.vnet.ibm.com&gt; --- src/util/network.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) Index: libvirt-acl/src/util/network.c =================================================================== --- libvirt-acl.orig/src/util/network.c +++ libvirt-acl/src/util/network.c @@ -1156,8 +1156,8 @@ virBandwidthEnable(virBandwidthPtr bandw virCommandFree(cmd); cmd = virCommandNew(TC); - virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", - "1:", "classid", "1:1", "htb", NULL); + virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", + "1:", "classid", "1:1", "htb", NULL); virCommandAddArgList(cmd, "rate", average, NULL);

@@ -1202,7 +1202,7 @@ virBandwidthEnable(virBandwidthPtr bandw virCommandAddArgList(cmd, "filter", "add", "dev", iface, "parent", "ffff:", "protocol", "ip", "u32", "match", "ip", "src", "0.0.0.0/0", "police", "rate", average, - "burst", burst, "mtu", burst, "drop", "flowid", + "burst", burst, "mtu", "2kb", "drop", "flowid", ":1", NULL);

On 08/10/2011 11:20 AM, Eric Blake wrote: > > On 08/10/2011 09:04 AM, Stefan Berger wrote: >> >> On 08/10/2011 02:55 AM, Upendra Moturi wrote: >> >> Hi, >> >> does this patch solve your problem? I am setting the MTU to fixed 2kb. >> Doing tests with scp seems to indicate that this improved the situation >> -- at least for me. >> >> Stefan >> >> Signed-off-by: Stefan Berger <stefanb(a)linux.vnet.ibm.com&gt; >> >> --- >> src/util/network.c | 16 ++++++++-------- >> 1 file changed, 8 insertions(+), 8 deletions(-) >> >> Index: libvirt-acl/src/util/network.c >> =================================================================== >> --- libvirt-acl.orig/src/util/network.c >> +++ libvirt-acl/src/util/network.c >> @@ -1156,8 +1156,8 @@ virBandwidthEnable(virBandwidthPtr bandw >> >> virCommandFree(cmd); >> cmd = virCommandNew(TC); >> - virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", >> - "1:", "classid", "1:1", "htb", NULL); >> + virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", >> + "1:", "classid", "1:1", "htb", NULL); >> virCommandAddArgList(cmd, "rate", average, NULL); > > This hunk was just indentation; you should push the whitespace changes as > a separate commit (and can push that now, under the trivial rule, without > needing further review).  [It doesn't help matters that thunderbird botched > the whitespace in my reply] > >> @@ -1202,7 +1202,7 @@ virBandwidthEnable(virBandwidthPtr bandw >> virCommandAddArgList(cmd, "filter", "add", "dev", iface, "parent", >> "ffff:", "protocol", "ip", "u32", "match", "ip", >> "src", "0.0.0.0/0", "police", "rate", average, >> - "burst", burst, "mtu", burst, "drop", "flowid", >> + "burst", burst, "mtu", "2kb", "drop", "flowid", >> ":1", NULL); > > Here's the meat of the proposed patch.  Is 2kb always valid, or will we > run into problems on NICs that insist on traditional limits near the 1518 > mark?  Should it be user-configurable?  I'm reluctant to ACK this without a > bit more understanding of why 2kb works, as well as feedback on test results > with the patch in place. > Yes, I am not an exper on tc. I have seen examples with the mtu being set to '1' for dropping icmp traffic for example. So if a VM produces smaller than 2kb packets, which on traditional networks it probably should, then the packets should go through. We could also leave out the mtu parameter above and it would go to 2kb by default. Check via tc filter show dev <ifname> root The 2kb parameter doesn't explain why 2MB doesn't work, though. Even if this wasn't the final fix, it at least 'improves the situation' (for me).    Stefan

On 08/11/2011 08:07 AM, Upendra Moturi wrote: > Thanks for your reply. > > The patch it appears still has problem. > I have changed only mtu value as 2kb instead of burst value in the > network.c file > > With average as 2048 , burst as 2048 and peak 2048 > scp copied at 30-40KB/s All test I did were with the patch applied. I saw the throughput collapsing yesterday at 2Mb mtu.

I generated a file using dd if=/dev/urandom of=testfile bs=1024 count=409600 I did the same test you did with 2048 without specifying a specific model of ethernet card and reach 1.9MB/s using scp'ing towards remote machine. When specifying <model type='virtio'/> I also get 1.9MB/s as result. > With average as 4096 , burst as 4096 and peak 4096 > scp copied at same 30-40KB/s > With these numbers and using the virtio model I ended up getting a throughput of 3.8Mb/s. The host is a 3.0.0 Linux kernel. Here's the command line output of the tc command inspecting the tap interface of the VM (vnet0). # tc filter show dev vnet0 root filter parent ffff: protocol ip pref 49152 u32 filter parent ffff: protocol ip pref 49152 u32 fh 800: ht divisor 1 filter parent ffff: protocol ip pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 match 00000000/00000000 at 12 police 0x5b rate 32768Kbit burst 4Mb mtu 2Kb action drop overhead 0b ref 1 bind 1 Maybe someone else can verify as well. Stefan > where as earlier it was 1 MB/s.(without patch) > > On Wed, Aug 10, 2011 at 9:16 PM, Stefan Berger > <stefanb(a)linux.vnet.ibm.com&gt; wrote: >> On 08/10/2011 11:20 AM, Eric Blake wrote: >>> On 08/10/2011 09:04 AM, Stefan Berger wrote: >>>> On 08/10/2011 02:55 AM, Upendra Moturi wrote: >>>> >>>> Hi, >>>> >>>> does this patch solve your problem? I am setting the MTU to fixed >>>> 2kb. >>>> Doing tests with scp seems to indicate that this improved the >>>> situation >>>> -- at least for me. >>>> >>>> Stefan >>>> >>>> Signed-off-by: Stefan Berger&lt;stefanb(a)linux.vnet.ibm.com&gt; >>>> >>>> --- >>>> src/util/network.c | 16 ++++++++-------- >>>> 1 file changed, 8 insertions(+), 8 deletions(-) >>>> >>>> Index: libvirt-acl/src/util/network.c >>>> =================================================================== >>>> --- libvirt-acl.orig/src/util/network.c >>>> +++ libvirt-acl/src/util/network.c >>>> @@ -1156,8 +1156,8 @@ virBandwidthEnable(virBandwidthPtr bandw >>>> >>>> virCommandFree(cmd); >>>> cmd = virCommandNew(TC); >>>> - virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", >>>> - "1:", "classid", "1:1", "htb", NULL); >>>> + virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", >>>> + "1:", "classid", "1:1", "htb", NULL); >>>> virCommandAddArgList(cmd, "rate", average, NULL); >>> This hunk was just indentation; you should push the whitespace >>> changes as >>> a separate commit (and can push that now, under the trivial rule, >>> without >>> needing further review). [It doesn't help matters that thunderbird >>> botched >>> the whitespace in my reply] >>> >>>> @@ -1202,7 +1202,7 @@ virBandwidthEnable(virBandwidthPtr bandw >>>> virCommandAddArgList(cmd, "filter", "add", "dev", iface, "parent", >>>> "ffff:", "protocol", "ip", "u32", "match", "ip", >>>> "src", "0.0.0.0/0", "police", "rate", average, >>>> - "burst", burst, "mtu", burst, "drop", "flowid", >>>> + "burst", burst, "mtu", "2kb", "drop", "flowid", >>>> ":1", NULL); >>> Here's the meat of the proposed patch. Is 2kb always valid, or >>> will we >>> run into problems on NICs that insist on traditional limits near >>> the 1518 >>> mark? Should it be user-configurable? I'm reluctant to ACK this >>> without a >>> bit more understanding of why 2kb works, as well as feedback on >>> test results >>> with the patch in place. >>> >> Yes, I am not an exper on tc. I have seen examples with the mtu >> being set to >> '1' for dropping icmp traffic for example. So if a VM produces >> smaller than >> 2kb packets, which on traditional networks it probably should, then the >> packets should go through. We could also leave out the mtu parameter >> above >> and it would go to 2kb by default. Check via >> >> tc filter show dev<ifname> root >> >> The 2kb parameter doesn't explain why 2MB doesn't work, though. >> Even if this wasn't the final fix, it at least 'improves the >> situation' (for >> me). >> >> Stefan >> >> >> > > -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

Thanks for your clarification. I am using 2.6.38-8-server kernel. Will test with 3.0.0 kernel

On Thu, Aug 11, 2011 at 7:18 PM, Stefan Berger <stefanb(a)linux.vnet.ibm.com&gt; wrote: > On 08/11/2011 09:39 AM, Stefan Berger wrote: >> On 08/11/2011 08:07 AM, Upendra Moturi wrote: >>> Thanks for your reply. >>> >>> The patch it appears still has problem. >>> I have changed only mtu value as 2kb instead of burst value in the >>> network.c file >>> >>> With average as 2048 , burst as 2048 and peak 2048 >>> scp copied at 30-40KB/s >> All test I did were with the patch applied. I saw the throughput >> collapsing yesterday at 2Mb mtu. > Let me clarify : Yesterday I saw the throughput collapse without the patch > applied. > > Stefan > >> I generated a file using >> >> dd if=/dev/urandom of=testfile bs=1024 count=409600 >> >> I did the same test you did with 2048 without specifying a specific model >> of ethernet card and reach 1.9MB/s using scp'ing towards remote machine. >> When specifying<model type='virtio'/> I also get 1.9MB/s as result. >>> With average as 4096 , burst as 4096 and peak 4096 >>> scp copied at same 30-40KB/s >>> >> With these numbers and using the virtio model I ended up getting a >> throughput of 3.8Mb/s. >> >> The host is a 3.0.0 Linux kernel. Here's the command line output of the tc >> command inspecting the tap interface of the VM (vnet0). >> >> # tc filter show dev vnet0 root >> filter parent ffff: protocol ip pref 49152 u32 >> filter parent ffff: protocol ip pref 49152 u32 fh 800: ht divisor 1 >> filter parent ffff: protocol ip pref 49152 u32 fh 800::800 order 2048 key >> ht 800 bkt 0 flowid :1 >> match 00000000/00000000 at 12 >> police 0x5b rate 32768Kbit burst 4Mb mtu 2Kb action drop overhead 0b >> ref 1 bind 1 >> >> >> Maybe someone else can verify as well. >> >> Stefan >> >>> where as earlier it was 1 MB/s.(without patch) >>> >>> On Wed, Aug 10, 2011 at 9:16 PM, Stefan Berger >>> <stefanb(a)linux.vnet.ibm.com&gt; wrote: >>>> On 08/10/2011 11:20 AM, Eric Blake wrote: >>>>> On 08/10/2011 09:04 AM, Stefan Berger wrote: >>>>>> On 08/10/2011 02:55 AM, Upendra Moturi wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> does this patch solve your problem? I am setting the MTU to fixed 2kb. >>>>>> Doing tests with scp seems to indicate that this improved the >>>>>> situation >>>>>> -- at least for me. >>>>>> >>>>>> Stefan >>>>>> >>>>>> Signed-off-by: Stefan Berger&lt;stefanb(a)linux.vnet.ibm.com&gt; >>>>>> >>>>>> --- >>>>>> src/util/network.c | 16 ++++++++-------- >>>>>> 1 file changed, 8 insertions(+), 8 deletions(-) >>>>>> >>>>>> Index: libvirt-acl/src/util/network.c >>>>>> =================================================================== >>>>>> --- libvirt-acl.orig/src/util/network.c >>>>>> +++ libvirt-acl/src/util/network.c >>>>>> @@ -1156,8 +1156,8 @@ virBandwidthEnable(virBandwidthPtr bandw >>>>>> >>>>>> virCommandFree(cmd); >>>>>> cmd = virCommandNew(TC); >>>>>> - virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", >>>>>> - "1:", "classid", "1:1", "htb", NULL); >>>>>> + virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent", >>>>>> + "1:", "classid", "1:1", "htb", NULL); >>>>>> virCommandAddArgList(cmd, "rate", average, NULL); >>>>> This hunk was just indentation; you should push the whitespace changes >>>>> as >>>>> a separate commit (and can push that now, under the trivial rule, >>>>> without >>>>> needing further review). [It doesn't help matters that thunderbird >>>>> botched >>>>> the whitespace in my reply] >>>>> >>>>>> @@ -1202,7 +1202,7 @@ virBandwidthEnable(virBandwidthPtr bandw >>>>>> virCommandAddArgList(cmd, "filter", "add", "dev", iface, "parent", >>>>>> "ffff:", "protocol", "ip", "u32", "match", "ip", >>>>>> "src", "0.0.0.0/0", "police", "rate", average, >>>>>> - "burst", burst, "mtu", burst, "drop", "flowid", >>>>>> + "burst", burst, "mtu", "2kb", "drop", "flowid", >>>>>> ":1", NULL); >>>>> Here's the meat of the proposed patch. Is 2kb always valid, or will we >>>>> run into problems on NICs that insist on traditional limits near the >>>>> 1518 >>>>> mark? Should it be user-configurable? I'm reluctant to ACK this >>>>> without a >>>>> bit more understanding of why 2kb works, as well as feedback on test >>>>> results >>>>> with the patch in place. >>>>> >>>> Yes, I am not an exper on tc. I have seen examples with the mtu being >>>> set to >>>> '1' for dropping icmp traffic for example. So if a VM produces smaller >>>> than >>>> 2kb packets, which on traditional networks it probably should, then the >>>> packets should go through. We could also leave out the mtu parameter >>>> above >>>> and it would go to 2kb by default. Check via >>>> >>>> tc filter show dev<ifname> root >>>> >>>> The 2kb parameter doesn't explain why 2MB doesn't work, though. >>>> Even if this wasn't the final fix, it at least 'improves the situation' >>>> (for >>>> me). >>>> >>>> Stefan >>>> >>>> >>>> >>> >> -- >> libvir-list mailing list >> libvir-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/libvir-list > -- > libvir-list mailing list > libvir-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list >