Re: [libvirt] Qemu/KVM guest boots 2x slower with vhost_net

On 10/05/11 01:12, Reeted wrote: Just a small update, Firstly: I cannot reproduce any slowness after boot by doing: # time /etc/init.d/chrony restart Restarting time daemon: Starting /usr/sbin/chronyd... chronyd is running and online. real 0m3.022s user 0m0.000s sys 0m0.000s since this is a network service I expected it to show the problem, but it doesn't. It takes exactly same time with and without vhost_net. Secondly, vhost_net appears to work correctly, because I have performed a NPtcp performance test between two guests in the same host, and these are the results: vhost_net deactivated for both hosts: NPtcp -h 192.168.7.81 Send and receive buffers are 16384 and 87380 bytes (A bug in Linux doubles the requested buffer sizes) Now starting the main loop 0: 1 bytes 917 times --> 0.08 Mbps in 92.07 usec 1: 2 bytes 1086 times --> 0.18 Mbps in 86.04 usec 2: 3 bytes 1162 times --> 0.27 Mbps in 85.08 usec 3: 4 bytes 783 times --> 0.36 Mbps in 85.34 usec 4: 6 bytes 878 times --> 0.54 Mbps in 85.42 usec 5: 8 bytes 585 times --> 0.72 Mbps in 85.31 usec 6: 12 bytes 732 times --> 1.07 Mbps in 85.52 usec 7: 13 bytes 487 times --> 1.16 Mbps in 85.52 usec 8: 16 bytes 539 times --> 1.43 Mbps in 85.26 usec 9: 19 bytes 659 times --> 1.70 Mbps in 85.43 usec 10: 21 bytes 739 times --> 1.77 Mbps in 90.71 usec 11: 24 bytes 734 times --> 2.13 Mbps in 86.13 usec 12: 27 bytes 822 times --> 2.22 Mbps in 92.80 usec 13: 29 bytes 478 times --> 2.35 Mbps in 94.02 usec 14: 32 bytes 513 times --> 2.60 Mbps in 93.75 usec 15: 35 bytes 566 times --> 3.15 Mbps in 84.77 usec 16: 45 bytes 674 times --> 4.01 Mbps in 85.56 usec 17: 48 bytes 779 times --> 4.32 Mbps in 84.70 usec 18: 51 bytes 811 times --> 4.61 Mbps in 84.32 usec 19: 61 bytes 465 times --> 5.08 Mbps in 91.57 usec 20: 64 bytes 537 times --> 5.22 Mbps in 93.46 usec 21: 67 bytes 551 times --> 5.73 Mbps in 89.20 usec 22: 93 bytes 602 times --> 8.28 Mbps in 85.73 usec 23: 96 bytes 777 times --> 8.45 Mbps in 86.70 usec 24: 99 bytes 780 times --> 8.71 Mbps in 86.72 usec 25: 125 bytes 419 times --> 11.06 Mbps in 86.25 usec 26: 128 bytes 575 times --> 11.38 Mbps in 85.80 usec 27: 131 bytes 591 times --> 11.60 Mbps in 86.17 usec 28: 189 bytes 602 times --> 16.55 Mbps in 87.14 usec 29: 192 bytes 765 times --> 16.80 Mbps in 87.19 usec 30: 195 bytes 770 times --> 17.11 Mbps in 86.94 usec 31: 253 bytes 401 times --> 22.04 Mbps in 87.59 usec 32: 256 bytes 568 times --> 22.64 Mbps in 86.25 usec 33: 259 bytes 584 times --> 22.68 Mbps in 87.12 usec 34: 381 bytes 585 times --> 33.19 Mbps in 87.58 usec 35: 384 bytes 761 times --> 33.54 Mbps in 87.36 usec 36: 387 bytes 766 times --> 33.91 Mbps in 87.08 usec 37: 509 bytes 391 times --> 44.23 Mbps in 87.80 usec 38: 512 bytes 568 times --> 44.70 Mbps in 87.39 usec 39: 515 bytes 574 times --> 45.21 Mbps in 86.90 usec 40: 765 bytes 580 times --> 66.05 Mbps in 88.36 usec 41: 768 bytes 754 times --> 66.73 Mbps in 87.81 usec 42: 771 bytes 760 times --> 67.02 Mbps in 87.77 usec 43: 1021 bytes 384 times --> 88.04 Mbps in 88.48 usec 44: 1024 bytes 564 times --> 88.30 Mbps in 88.48 usec 45: 1027 bytes 566 times --> 88.63 Mbps in 88.40 usec 46: 1533 bytes 568 times --> 71.75 Mbps in 163.00 usec 47: 1536 bytes 408 times --> 72.11 Mbps in 162.51 usec 48: 1539 bytes 410 times --> 71.71 Mbps in 163.75 usec 49: 2045 bytes 204 times --> 95.40 Mbps in 163.55 usec 50: 2048 bytes 305 times --> 95.26 Mbps in 164.02 usec 51: 2051 bytes 305 times --> 95.33 Mbps in 164.14 usec 52: 3069 bytes 305 times --> 141.16 Mbps in 165.87 usec 53: 3072 bytes 401 times --> 142.19 Mbps in 164.83 usec 54: 3075 bytes 404 times --> 150.68 Mbps in 155.70 usec 55: 4093 bytes 214 times --> 192.36 Mbps in 162.33 usec 56: 4096 bytes 307 times --> 193.21 Mbps in 161.74 usec 57: 4099 bytes 309 times --> 213.24 Mbps in 146.66 usec 58: 6141 bytes 341 times --> 330.80 Mbps in 141.63 usec 59: 6144 bytes 470 times --> 328.09 Mbps in 142.87 usec 60: 6147 bytes 466 times --> 330.53 Mbps in 141.89 usec 61: 8189 bytes 235 times --> 437.29 Mbps in 142.87 usec 62: 8192 bytes 349 times --> 436.23 Mbps in 143.27 usec 63: 8195 bytes 349 times --> 436.99 Mbps in 143.08 usec 64: 12285 bytes 349 times --> 625.88 Mbps in 149.75 usec 65: 12288 bytes 445 times --> 626.27 Mbps in 149.70 usec 66: 12291 bytes 445 times --> 626.15 Mbps in 149.76 usec 67: 16381 bytes 222 times --> 793.58 Mbps in 157.48 usec 68: 16384 bytes 317 times --> 806.90 Mbps in 154.91 usec 69: 16387 bytes 322 times --> 796.81 Mbps in 156.90 usec 70: 24573 bytes 318 times --> 1127.58 Mbps in 166.26 usec 71: 24576 bytes 400 times --> 1125.20 Mbps in 166.64 usec 72: 24579 bytes 400 times --> 1124.84 Mbps in 166.71 usec 73: 32765 bytes 200 times --> 1383.86 Mbps in 180.64 usec 74: 32768 bytes 276 times --> 1376.05 Mbps in 181.68 usec 75: 32771 bytes 275 times --> 1377.47 Mbps in 181.51 usec 76: 49149 bytes 275 times --> 1824.90 Mbps in 205.48 usec 77: 49152 bytes 324 times --> 1813.95 Mbps in 206.73 usec 78: 49155 bytes 322 times --> 1765.68 Mbps in 212.40 usec 79: 65533 bytes 156 times --> 2193.44 Mbps in 227.94 usec 80: 65536 bytes 219 times --> 2186.79 Mbps in 228.65 usec 81: 65539 bytes 218 times --> 2186.98 Mbps in 228.64 usec 82: 98301 bytes 218 times --> 2831.01 Mbps in 264.92 usec 83: 98304 bytes 251 times --> 2804.76 Mbps in 267.40 usec 84: 98307 bytes 249 times --> 2824.62 Mbps in 265.53 usec 85: 131069 bytes 125 times --> 3106.48 Mbps in 321.90 usec 86: 131072 bytes 155 times --> 3033.71 Mbps in 329.63 usec 87: 131075 bytes 151 times --> 3044.89 Mbps in 328.43 usec 88: 196605 bytes 152 times --> 4196.94 Mbps in 357.40 usec 89: 196608 bytes 186 times --> 4358.25 Mbps in 344.17 usec 90: 196611 bytes 193 times --> 4362.34 Mbps in 343.86 usec 91: 262141 bytes 96 times --> 4654.49 Mbps in 429.69 usec 92: 262144 bytes 116 times --> 4727.16 Mbps in 423.09 usec 93: 262147 bytes 118 times --> 4697.22 Mbps in 425.79 usec 94: 393213 bytes 117 times --> 5452.51 Mbps in 550.20 usec 95: 393216 bytes 121 times --> 5360.27 Mbps in 559.67 usec 96: 393219 bytes 119 times --> 5358.03 Mbps in 559.91 usec 97: 524285 bytes 59 times --> 5053.83 Mbps in 791.47 usec 98: 524288 bytes 63 times --> 5033.86 Mbps in 794.62 usec 99: 524291 bytes 62 times --> 5691.44 Mbps in 702.81 usec 100: 786429 bytes 71 times --> 5750.68 Mbps in 1043.35 usec 101: 786432 bytes 63 times --> 5809.21 Mbps in 1032.84 usec 102: 786435 bytes 64 times --> 5864.45 Mbps in 1023.12 usec 103: 1048573 bytes 32 times --> 5755.24 Mbps in 1390.03 usec 104: 1048576 bytes 35 times --> 6001.51 Mbps in 1333.00 usec 105: 1048579 bytes 37 times --> 6099.40 Mbps in 1311.61 usec 106: 1572861 bytes 38 times --> 6061.69 Mbps in 1979.64 usec 107: 1572864 bytes 33 times --> 6144.15 Mbps in 1953.08 usec 108: 1572867 bytes 34 times --> 6108.20 Mbps in 1964.58 usec 109: 2097149 bytes 16 times --> 6128.72 Mbps in 2610.65 usec 110: 2097152 bytes 19 times --> 6271.35 Mbps in 2551.29 usec 111: 2097155 bytes 19 times --> 6273.55 Mbps in 2550.39 usec 112: 3145725 bytes 19 times --> 6146.28 Mbps in 3904.79 usec 113: 3145728 bytes 17 times --> 6288.29 Mbps in 3816.62 usec 114: 3145731 bytes 17 times --> 6234.73 Mbps in 3849.41 usec 115: 4194301 bytes 8 times --> 5852.76 Mbps in 5467.50 usec 116: 4194304 bytes 9 times --> 5886.74 Mbps in 5435.94 usec 117: 4194307 bytes 9 times --> 5887.35 Mbps in 5435.39 usec 118: 6291453 bytes 9 times --> 4502.11 Mbps in 10661.67 usec 119: 6291456 bytes 6 times --> 4541.26 Mbps in 10569.75 usec 120: 6291459 bytes 6 times --> 4465.98 Mbps in 10747.93 usec 121: 8388605 bytes 3 times --> 4601.84 Mbps in 13907.47 usec 122: 8388608 bytes 3 times --> 4590.50 Mbps in 13941.84 usec 123: 8388611 bytes 3 times --> 4195.17 Mbps in 15255.65 usec vhost_net activated for both hosts: NPtcp -h 192.168.7.81 Send and receive buffers are 16384 and 87380 bytes (A bug in Linux doubles the requested buffer sizes) Now starting the main loop 0: 1 bytes 1013 times --> 0.10 Mbps in 75.89 usec 1: 2 bytes 1317 times --> 0.21 Mbps in 74.03 usec 2: 3 bytes 1350 times --> 0.30 Mbps in 76.90 usec 3: 4 bytes 866 times --> 0.43 Mbps in 71.27 usec 4: 6 bytes 1052 times --> 0.60 Mbps in 76.02 usec 5: 8 bytes 657 times --> 0.79 Mbps in 76.88 usec 6: 12 bytes 812 times --> 1.24 Mbps in 73.72 usec 7: 13 bytes 565 times --> 1.40 Mbps in 70.60 usec 8: 16 bytes 653 times --> 1.58 Mbps in 77.05 usec 9: 19 bytes 730 times --> 1.90 Mbps in 76.25 usec 10: 21 bytes 828 times --> 1.98 Mbps in 80.85 usec 11: 24 bytes 824 times --> 2.47 Mbps in 74.22 usec 12: 27 bytes 954 times --> 2.73 Mbps in 75.45 usec 13: 29 bytes 589 times --> 3.06 Mbps in 72.23 usec 14: 32 bytes 668 times --> 3.26 Mbps in 74.84 usec 15: 35 bytes 709 times --> 3.46 Mbps in 77.09 usec 16: 45 bytes 741 times --> 4.50 Mbps in 76.35 usec 17: 48 bytes 873 times --> 4.83 Mbps in 75.90 usec 18: 51 bytes 905 times --> 5.50 Mbps in 70.72 usec 19: 61 bytes 554 times --> 6.36 Mbps in 73.14 usec 20: 64 bytes 672 times --> 6.28 Mbps in 77.77 usec 21: 67 bytes 663 times --> 6.39 Mbps in 80.06 usec 22: 93 bytes 671 times --> 9.44 Mbps in 75.15 usec 23: 96 bytes 887 times --> 9.52 Mbps in 76.90 usec 24: 99 bytes 880 times --> 10.55 Mbps in 71.57 usec 25: 125 bytes 508 times --> 12.63 Mbps in 75.49 usec 26: 128 bytes 657 times --> 12.30 Mbps in 79.38 usec 27: 131 bytes 639 times --> 12.72 Mbps in 78.57 usec 28: 189 bytes 660 times --> 18.36 Mbps in 78.55 usec 29: 192 bytes 848 times --> 18.84 Mbps in 77.75 usec 30: 195 bytes 864 times --> 18.91 Mbps in 78.69 usec 31: 253 bytes 443 times --> 24.04 Mbps in 80.28 usec 32: 256 bytes 620 times --> 26.61 Mbps in 73.40 usec 33: 259 bytes 686 times --> 26.09 Mbps in 75.75 usec 34: 381 bytes 672 times --> 40.04 Mbps in 72.59 usec 35: 384 bytes 918 times --> 39.67 Mbps in 73.86 usec 36: 387 bytes 906 times --> 40.68 Mbps in 72.58 usec 37: 509 bytes 469 times --> 51.70 Mbps in 75.11 usec 38: 512 bytes 664 times --> 51.55 Mbps in 75.77 usec 39: 515 bytes 662 times --> 49.61 Mbps in 79.19 usec 40: 765 bytes 637 times --> 75.91 Mbps in 76.89 usec 41: 768 bytes 867 times --> 76.03 Mbps in 77.07 usec 42: 771 bytes 866 times --> 76.21 Mbps in 77.19 usec 43: 1021 bytes 436 times --> 99.46 Mbps in 78.32 usec 44: 1024 bytes 637 times --> 100.04 Mbps in 78.10 usec 45: 1027 bytes 641 times --> 100.06 Mbps in 78.31 usec 46: 1533 bytes 641 times --> 113.15 Mbps in 103.36 usec 47: 1536 bytes 644 times --> 127.72 Mbps in 91.75 usec 48: 1539 bytes 727 times --> 102.87 Mbps in 114.14 usec 49: 2045 bytes 293 times --> 177.68 Mbps in 87.81 usec 50: 2048 bytes 569 times --> 103.58 Mbps in 150.85 usec 51: 2051 bytes 331 times --> 107.53 Mbps in 145.52 usec 52: 3069 bytes 344 times --> 204.05 Mbps in 114.75 usec 53: 3072 bytes 580 times --> 207.53 Mbps in 112.93 usec 54: 3075 bytes 590 times --> 211.37 Mbps in 110.99 usec 55: 4093 bytes 301 times --> 285.23 Mbps in 109.48 usec 56: 4096 bytes 456 times --> 317.27 Mbps in 98.50 usec 57: 4099 bytes 507 times --> 332.92 Mbps in 93.93 usec 58: 6141 bytes 532 times --> 462.96 Mbps in 101.20 usec 59: 6144 bytes 658 times --> 451.75 Mbps in 103.76 usec 60: 6147 bytes 642 times --> 478.19 Mbps in 98.07 usec 61: 8189 bytes 340 times --> 743.81 Mbps in 84.00 usec 62: 8192 bytes 595 times --> 695.89 Mbps in 89.81 usec 63: 8195 bytes 556 times --> 702.95 Mbps in 88.94 usec 64: 12285 bytes 562 times --> 945.94 Mbps in 99.08 usec 65: 12288 bytes 672 times --> 870.86 Mbps in 107.65 usec 66: 12291 bytes 619 times --> 954.94 Mbps in 98.20 usec 67: 16381 bytes 339 times --> 1003.02 Mbps in 124.60 usec 68: 16384 bytes 401 times --> 652.84 Mbps in 191.47 usec 69: 16387 bytes 261 times --> 872.02 Mbps in 143.37 usec 70: 24573 bytes 348 times --> 1105.61 Mbps in 169.57 usec 71: 24576 bytes 393 times --> 1037.52 Mbps in 180.72 usec 72: 24579 bytes 368 times --> 1066.39 Mbps in 175.85 usec 73: 32765 bytes 189 times --> 1271.24 Mbps in 196.64 usec 74: 32768 bytes 254 times --> 1253.73 Mbps in 199.41 usec 75: 32771 bytes 250 times --> 1101.71 Mbps in 226.94 usec 76: 49149 bytes 220 times --> 1704.99 Mbps in 219.93 usec 77: 49152 bytes 303 times --> 1678.17 Mbps in 223.46 usec 78: 49155 bytes 298 times --> 1648.32 Mbps in 227.52 usec 79: 65533 bytes 146 times --> 1940.36 Mbps in 257.67 usec 80: 65536 bytes 194 times --> 1785.37 Mbps in 280.05 usec 81: 65539 bytes 178 times --> 2079.85 Mbps in 240.41 usec 82: 98301 bytes 207 times --> 2840.36 Mbps in 264.04 usec 83: 98304 bytes 252 times --> 3441.30 Mbps in 217.94 usec 84: 98307 bytes 305 times --> 3575.33 Mbps in 209.78 usec 85: 131069 bytes 158 times --> 3145.83 Mbps in 317.87 usec 86: 131072 bytes 157 times --> 3283.65 Mbps in 304.54 usec 87: 131075 bytes 164 times --> 3610.07 Mbps in 277.01 usec 88: 196605 bytes 180 times --> 4921.05 Mbps in 304.81 usec 89: 196608 bytes 218 times --> 4953.98 Mbps in 302.79 usec 90: 196611 bytes 220 times --> 4841.76 Mbps in 309.81 usec 91: 262141 bytes 107 times --> 4546.37 Mbps in 439.91 usec 92: 262144 bytes 113 times --> 4730.30 Mbps in 422.81 usec 93: 262147 bytes 118 times --> 5211.50 Mbps in 383.77 usec 94: 393213 bytes 130 times --> 7191.67 Mbps in 417.15 usec 95: 393216 bytes 159 times --> 7423.89 Mbps in 404.10 usec 96: 393219 bytes 164 times --> 7321.70 Mbps in 409.74 usec 97: 524285 bytes 81 times --> 7631.75 Mbps in 524.12 usec 98: 524288 bytes 95 times --> 7287.79 Mbps in 548.86 usec 99: 524291 bytes 91 times --> 7253.28 Mbps in 551.48 usec 100: 786429 bytes 90 times --> 8451.33 Mbps in 709.94 usec 101: 786432 bytes 93 times --> 8755.43 Mbps in 685.29 usec 102: 786435 bytes 97 times --> 8740.15 Mbps in 686.49 usec 103: 1048573 bytes 48 times --> 9220.97 Mbps in 867.59 usec 104: 1048576 bytes 57 times --> 8512.15 Mbps in 939.83 usec 105: 1048579 bytes 53 times --> 8556.70 Mbps in 934.94 usec 106: 1572861 bytes 53 times --> 9566.40 Mbps in 1254.39 usec 107: 1572864 bytes 53 times --> 10165.18 Mbps in 1180.50 usec 108: 1572867 bytes 56 times --> 11420.63 Mbps in 1050.73 usec 109: 2097149 bytes 31 times --> 11295.29 Mbps in 1416.52 usec 110: 2097152 bytes 35 times --> 11869.30 Mbps in 1348.02 usec 111: 2097155 bytes 37 times --> 11407.22 Mbps in 1402.62 usec 112: 3145725 bytes 35 times --> 12821.47 Mbps in 1871.86 usec 113: 3145728 bytes 35 times --> 11727.57 Mbps in 2046.46 usec 114: 3145731 bytes 32 times --> 12803.10 Mbps in 1874.55 usec 115: 4194301 bytes 17 times --> 10009.28 Mbps in 3197.03 usec 116: 4194304 bytes 15 times --> 10283.54 Mbps in 3111.77 usec 117: 4194307 bytes 16 times --> 10923.95 Mbps in 2929.34 usec 118: 6291453 bytes 17 times --> 11959.10 Mbps in 4013.68 usec 119: 6291456 bytes 16 times --> 10674.76 Mbps in 4496.59 usec 120: 6291459 bytes 14 times --> 10868.07 Mbps in 4416.61 usec 121: 8388605 bytes 7 times --> 9456.16 Mbps in 6768.07 usec 122: 8388608 bytes 7 times --> 9303.58 Mbps in 6879.07 usec 123: 8388611 bytes 7 times --> 10048.79 Mbps in 6368.93 usec so the case vhost_net is indeed faster in NPtcp by a factor almost 2, but that's only visible for the very high speeds and buffer sizes. Also note that the speed in the vhost_net case is much more variable, even if it's higher on average. I expected more difference... Actually I expected no-vhost to be slower than what it is. Kudos to the developers. If you have any more ideas for the slower boot please tell. I am of course not worried about waiting 10-30 more seconds at boot time, I am worried that if there is some factor 2x or 3x slowness somewhere, that can bite me in production without me even realizing. And after having seen the above no-vhost_net tcp benchmarks I guess I don't really need vhost_net active for these VMs during production, so I will just disable vhost_net to be on the safe side until I can track down the boot-time slowness somehow. Thanks for your help R.