On Thu, Jul 27 at 14:05:09 +0200 Peter Krempa wrote:

On Thu, Jul 20, 2017 at 21:36:28 -0400, Laine Stump wrote:

...

On 07/18/2017 07:12 AM, Michal Privoznik wrote:

[...]

...

...

...

...

So only users that know how virtio works under the hood are expected to also know what rx/tx queue size is and how to set it. But frankly, I This statement is ridiculous by itself. Why? There are more experienced users (for whom libvirt's just a stable API) and less experienced ones (for whom libvirt's and tools on the top of it are great for their automatic chose of parameters, e.g. gnome boxes).

Beyond that, that same statement (or something nearly identical) is repeated in multiple places throughout the XML documentation. There are at least two classes of these attributes that I can think of:

1) attributes that are automatically set to a sane value by libvirt when not specified (and they usually *aren't* specified), and saved in the config XML in order to assure they are set the same every time the domain is started (to preserve guest ABI). These are intended to record whatever was the default setting for the attribute at the time the domain was created. For example, a pcie-root-port controller might have <model name='ioh3420'/> set, if that was the only type of pcie-root-port available at the time a domain was created; this comes in handy now that there is a generic pcie-root-port (which also has <model name='pcie-root-port'/>) - existing domains don't get their ABI screwed up when they're migrated to a newer host.

2) knobs that have been added in over the years at the request/demand from below (qemu) and above (ovirt / openstack), many of them obscure, difficult to explain with any amount of useful detail, and almost never worthy of being manually set (and if you "don't know what you're doing", you're just as likely to make things worse as to make them better).

tx_queue_size is one of the latter.

For either of these types of attributes, they need to be documented so that someone encountering them (or actively searching them out) will at least have a basic idea what the attribute is named / used for, but we also need to warn the casual user to not mess with them. I don't see anything ridiculous about that.

We, Ericsson, have been driving the possibility to increase the Rx and Tx queue sizes for virtio interfaces in Qemu because our throughput-intensive and packet-drop sensitive virtual network functions (VNFs) were suffering frequent early packet drops. In the age of DPDK and with packet rates in the Mpps range the hard-coded default queue size of 256 in Qemu was no longer adequate to compensate short interrupts on the receiving side or microbursts on the sending size. Many measurements across the industry had proven that increasing the queue size to 1024 very effectively reduced the packet drop probability and consequently improved throughput significantly. The Qemu community could not change the virtio-net default queue size to 1024 to avoid compatibility issues and limitations in legacy virtio drivers and virtio backends. But many VNFs and the DPDK vhostuser backend in vSwitches like OVS and fd.io/VPP can handle 1K queue size and benefit from it. So they made the queue sizes configurable in Qemu to give NFVI operators the chance to tune the performance of their infrastructure. This change in Libvirt is about making this tuning option available for Qemu users like OpenStack. There is no other way than Libvirt to make this accessible as Qemu does not have the facility of global default parameters.

...

...

...

...

think users setting this are always gonna go with the highest value avaliable (1024). Such detailed description is a copy of rx_virtio_queue size description which is result of review.

Reasons not to set to 1024 could include compatibility constraints (see above) or a trade-off between packet drop probability and maximum latency/jitter.

...

...

...

...

...

Is it really needed? How should it be configured? Can't we or qemu pick a sane value?

No. Some users need bigger virtio rings otherwise they see a packet drop. So this is a fine tuning that heavily depends on the use

case.

...

Thus libvirt should not try to come up with some value. That's why I think it's wrong. What's the drawback of setting it to maximum? Which workloads will hit it? Why is the default not sufficient?

And most notably, how do the users figure out that they need it?

In our case the "user" is the NFVI deployment tool chain with tuning input from NFV solution architects, guided by Installation/Dimensioning guides.

...

I think it's a bit too much burden on libvirt to expect that much detail to be included in formatdomain.html. Heck, I don't know if anyone even *knows* that much detail right now.

That proves my point kind of. If there isn't knowledge how to set up this, then why even add the option. Is there really a point?

I think we could improve the documentation a bit with reference to vhostuser interfaces as main target interface type and mentioning potential compatibility constraints for applications. We would actually like to see an additional global config knob in Libvirt to (optionally) set default Rx and Tx queue sizes for vhostuser interfaces, so that Libvirt consumers like OpenStack would not have to be modified to use 1K queues. BR, Jan