[libvirt] [PATCH v2] bandwidth: Adjust documentation
Recent autotest/virt-test testing on f20 discovered an anomaly in how the bandwidth options are documented and used. This was discovered due to a bug fix in the /sbin/tc utility found in iproute-3.11.0.1 (on f20) in which overflow was actually caught and returned as an error. The fix was first introduced in iproute-3.10 (search on iproute2 commit 'a303853e'). The autotest/virt-test test for virsh domiftune was attempting to send the largest unsigned integer value (4294967295) for maximum value testing. The libvirt xml implementation was designed to manage values in kilobytes thus when this value was passed to /sbin/tc, it (now) properly rejected the 4294967295kbps value. Investigation of the problem discovered that formatdomain.html.in and formatnetwork.html.in described the elements and property types slightly differently, although they use the same code - virNetDevBandwidthParseRate() (shared by portgroups, domains, and networks xml parsers). Rather than have the descriptions in two places, this patch will combine and reword the description under formatnetwork.html.in and have formatdomain.html.in link to that description. This documentation faux pas was continued into the virsh man page where the bandwidth description for both 'attach-interface' and 'domiftune' did not indicate the format of each value, thus leading to the test using largest unsigned integer value assuming "bps" rather than "kbps", which ultimately was wrong. Signed-off-by: John Ferlan <jferlan@redhat.com> --- Changes since v1: -> Reflection and some private IRC validation results in changing to use "kilobyte" and not "kibibyte" -> Although the smallest unit is 1 kb/kbps, it's felt that things aren't headed in the going smaller direction, rather things are desired to be larger. So the inability to "test" the maximum of 4294967295 bps in favor of 4294967 kbps shouldn't really matter. docs/formatdomain.html.in | 35 +++----------------- docs/formatnetwork.html.in | 81 ++++++++++++++++++++++++++++++++++------------ tools/virsh.pod | 10 ++++-- 3 files changed, 73 insertions(+), 53 deletions(-) diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 4660983..ea1a97b 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -3750,37 +3750,10 @@ qemu-kvm -net nic,model=? /dev/null <p> This part of interface XML provides setting quality of service. Incoming - and outgoing traffic can be shaped independently. The - <code>bandwidth</code> element can have at most one <code>inbound</code> - and at most one <code>outbound</code> child elements. Leaving any of these - children element out result in no QoS applied on that traffic direction. - So, when you want to shape only domain's incoming traffic, use - <code>inbound</code> only, and vice versa. Each of these elements have one - mandatory attribute <code>average</code> (or <code>floor</code> as - described below). <code>average</code> specifies average bit rate on - the interface being shaped. Then there are two optional attributes: - <code>peak</code>, which specifies maximum rate at which interface can send - data, and <code>burst</code>, amount of bytes that can be burst at - <code>peak</code> speed. Accepted values for attributes are integer - numbers. The units for <code>average</code> and <code>peak</code> attributes - are kilobytes per second, and for the <code>burst</code> just kilobytes. - Note the limitation of implementation: the <code>peak</code> attribute in - <code>outbound</code> element is ignored (as linux ingress filters don't - know it yet). <span class="since">Since 0.9.4</span> The <code>inbound</code> can - optionally have <code>floor</code> attribute. This is there for - guaranteeing minimal throughput for shaped interfaces. This, however, - requires that all traffic goes through one point where QoS decisions can - take place. That's why this attribute works only for virtual networks for - now (that is <code><interface type='network'/></code> with a - forward type of route, nat, or no forward at all). Moreover, the - virtual network the interface is connected to is required to have at least - inbound QoS set (<code>average</code> at least). Moreover, with - <code>floor</code> attribute users don't need to specify - <code>average</code>. However, <code>peak</code> and <code>burst</code> - attributes still require <code>average</code>. Currently, linux kernel - doesn't allow ingress qdiscs to have any classes therefore - <code>floor</code> can be applied only on <code>inbound</code> and not - <code>outbound</code>. <span class="since">Since 1.0.1</span> + and outgoing traffic can be shaped independently. + The <code>bandwidth</code> element and its child elements are described + in the <a href="formatnetwork.html#elementQoS">QoS</a> section of + the Network XML. </p> <h5><a name="elementVlanTag">Setting VLAN tag (on supported network types only)</a></h5> diff --git a/docs/formatnetwork.html.in b/docs/formatnetwork.html.in index 1ca1bec..d4c390a 100644 --- a/docs/formatnetwork.html.in +++ b/docs/formatnetwork.html.in @@ -412,40 +412,81 @@ <p> The <code><bandwidth></code> element allows setting - quality of service for a particular network. - <span class="since">Since 0.9.4</span> The limits specified + quality of service for a particular network + (<span class="since">since 0.9.4</span>). For a <code>domain</code> + object, the limits specified are applied to the domain traffic. + For a <code>network</code> object, the limits specified are applied to the aggregate of all traffic to/from all guest interfaces attached to that network, <b>not</b> to each guest - interface individually. Setting <code>bandwidth</code> for a + interface individually. Setting <code>bandwidth</code> for a network is supported only for networks with a <code><forward></code> mode of <code>route</code>, <code>nat</code>, or no mode at all - (i.e. an "isolated" network). <code>bandwidth</code> setting + (i.e. an "isolated" network). Setting <code>bandwidth</code> is <b>not</b> supported for forward modes of <code>bridge</code>, <code>passthrough</code>, <code>private</code>, or <code>hostdev</code>, and attempts to do this will lead to - a failure to define the network (or to create a transient - network). + a failure to define the network (or to create a transient network). </p> <p> Incoming and outgoing traffic can be shaped independently. The - <code>bandwidth</code> element can have at most - one <code>inbound</code> and at most one <code>outbound</code> + <code>bandwidth</code> element can have at most one + <code>inbound</code> and at most one <code>outbound</code> child element. Leaving either of these children elements out results in no QoS applied for that traffic direction. So, - when you want to shape only a network's incoming traffic, use + when you want to shape only incoming traffic, use <code>inbound</code> only, and vice versa. Each of these - elements have one mandatory attribute - <code>average</code>, - which specifies the desired average bit rate for the interface - being shaped (in kibibytes/second). There are also two - optional attributes: <code>peak</code>, which specifies the - maximum rate at which the bridge can send data (again in - kibibytes/second), and <code>burst</code> - the amount of - bytes that can be transmitted in a single burst at - <code>peak</code> speed (in kibibytes). Accepted values for - attributes are integer numbers. The allotted bandwidth is - shared equally between domains connected to the network. + elements have one mandatory attribute - <code>average</code> (or + <code>floor</code> as described below). The attributes are as follows, + where accepted values for each attribute is an integer number. </p> + <dl> + <dt><code>average</code></dt> + <dd> + Specifies the desired average bit rate for the interface + being shaped (in kilobytes/second). + </dd> + <dt><code>peak</code></dt> + <dd> + Optional attribute which specifies the maximum rate at + which the bridge can send data (in kilobytes/second). + Note the limitation of implementation: this attribute in the + <code>outbound</code> element is ignored (as Linux ingress + filters don't know it yet). + </dd> + <dt><code>burst</code></dt> + <dd> + Optional attribute which specifies the amount of kilobytes that + can be transmitted in a single burst at <code>peak</code> speed. + </dd> + <dt><code>floor</code></dt> + <dd> + Optional attribute available only for the <code>inbound</code> + element. This attribute guarantees minimal throughput for + shaped interfaces. This, however, requires that all traffic + goes through one point where QoS decisions can take place, hence + why this attribute works only for virtual networks for now + (that is <code><interface type='network'/></code> with a + forward type of route, nat, or no forward at all). Moreover, the + virtual network the interface is connected to is required to have + at least inbound QoS set (<code>average</code> at least). If + using the <code>floor</code> attribute users don't need to specify + <code>average</code>. However, <code>peak</code> and + <code>burst</code> attributes still require <code>average</code>. + Currently, the Linux kernel doesn't allow ingress qdiscs to have + any classes therefore <code>floor</code> can be applied only + on <code>inbound</code> and not <code>outbound</code>. + </dd> + </dl> + + <p> + Attributes <code>average</code>, <code>peak</code>, and + <code>burst</code> are available + <span class="since">since 0.9.4</span>, while the + <code>floor</code> attribute is available + <span class="since">since 1.0.1</span>. + </p> + <p> A <code><portgroup></code> element can also include a <code><bandwidth></code> element. In that case, the @@ -561,7 +602,7 @@ network), and each portgroup has a name, as well as various subelements associated with it. The currently supported subelements are <code><bandwidth></code> - (documented <a href="formatdomain.html#elementQoS">here</a>) + (described <a href="formatnetwork.html#elementQoS">here</a>) and <code><virtualport></code> (documented <a href="formatdomain.html#elementsNICSDirect">here</a>). If a domain interface definition specifies a portgroup (by diff --git a/tools/virsh.pod b/tools/virsh.pod index f221475..30dcbb8 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -680,7 +680,10 @@ or the MAC address. If no I<--inbound> or I<--outbound> is specified, this command will query and show the bandwidth settings. Otherwise, it will set the inbound or outbound bandwidth. I<average,peak,burst> is the same as -in command I<attach-interface>. +in command I<attach-interface>. Values for I<average> and I<peak> are +expressed in kilobytes per second, while I<burst> is expressed in kilobytes +in a single burst at -I<peak> speed as described in the Network XML +documentation at L<http://libvirt.org/formatnetwork.html#elementQoS>. If I<--live> is specified, affect a running guest. If I<--config> is specified, affect the next boot of a persistent guest. @@ -2019,7 +2022,10 @@ of the network interface. I<script> allows to specify a path to a script handling a bridge instead of the default one. I<model> allows to specify the model type. I<inbound> and I<outbound> control the bandwidth of the interface. I<peak> and I<burst> are optional, so "average,peak", "average,,burst" and -"average" are also legal. +"average" are also legal. Values for I<average> and I<peak> are +expressed in kilobytes per second, while I<burst> is expressed in kilobytes +in a single burst at -I<peak> speed as described in the Network XML +documentation at L<http://libvirt.org/formatnetwork.html#elementQoS>. If I<--live> is specified, affect a running domain. If I<--config> is specified, affect the next startup of a persistent domain. -- 1.8.4.2
On 13.02.2014 19:33, John Ferlan wrote:
Recent autotest/virt-test testing on f20 discovered an anomaly in how the bandwidth options are documented and used. This was discovered due to a bug fix in the /sbin/tc utility found in iproute-3.11.0.1 (on f20) in which overflow was actually caught and returned as an error. The fix was first introduced in iproute-3.10 (search on iproute2 commit 'a303853e').
The autotest/virt-test test for virsh domiftune was attempting to send the largest unsigned integer value (4294967295) for maximum value testing. The libvirt xml implementation was designed to manage values in kilobytes thus when this value was passed to /sbin/tc, it (now) properly rejected the 4294967295kbps value.
Investigation of the problem discovered that formatdomain.html.in and formatnetwork.html.in described the elements and property types slightly differently, although they use the same code - virNetDevBandwidthParseRate() (shared by portgroups, domains, and networks xml parsers). Rather than have the descriptions in two places, this patch will combine and reword the description under formatnetwork.html.in and have formatdomain.html.in link to that description.
This documentation faux pas was continued into the virsh man page where the bandwidth description for both 'attach-interface' and 'domiftune' did not indicate the format of each value, thus leading to the test using largest unsigned integer value assuming "bps" rather than "kbps", which ultimately was wrong.
Signed-off-by: John Ferlan <jferlan@redhat.com> ---
Changes since v1: -> Reflection and some private IRC validation results in changing to use "kilobyte" and not "kibibyte" -> Although the smallest unit is 1 kb/kbps, it's felt that things aren't headed in the going smaller direction, rather things are desired to be larger. So the inability to "test" the maximum of 4294967295 bps in favor of 4294967 kbps shouldn't really matter.
docs/formatdomain.html.in | 35 +++----------------- docs/formatnetwork.html.in | 81 ++++++++++++++++++++++++++++++++++------------ tools/virsh.pod | 10 ++++-- 3 files changed, 73 insertions(+), 53 deletions(-)
diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in index 4660983..ea1a97b 100644 --- a/docs/formatdomain.html.in +++ b/docs/formatdomain.html.in @@ -3750,37 +3750,10 @@ qemu-kvm -net nic,model=? /dev/null
<p> This part of interface XML provides setting quality of service. Incoming - and outgoing traffic can be shaped independently. The - <code>bandwidth</code> element can have at most one <code>inbound</code> - and at most one <code>outbound</code> child elements. Leaving any of these - children element out result in no QoS applied on that traffic direction. - So, when you want to shape only domain's incoming traffic, use - <code>inbound</code> only, and vice versa. Each of these elements have one - mandatory attribute <code>average</code> (or <code>floor</code> as - described below). <code>average</code> specifies average bit rate on - the interface being shaped. Then there are two optional attributes: - <code>peak</code>, which specifies maximum rate at which interface can send - data, and <code>burst</code>, amount of bytes that can be burst at - <code>peak</code> speed. Accepted values for attributes are integer - numbers. The units for <code>average</code> and <code>peak</code> attributes - are kilobytes per second, and for the <code>burst</code> just kilobytes. - Note the limitation of implementation: the <code>peak</code> attribute in - <code>outbound</code> element is ignored (as linux ingress filters don't - know it yet). <span class="since">Since 0.9.4</span> The <code>inbound</code> can - optionally have <code>floor</code> attribute. This is there for - guaranteeing minimal throughput for shaped interfaces. This, however, - requires that all traffic goes through one point where QoS decisions can - take place. That's why this attribute works only for virtual networks for - now (that is <code><interface type='network'/></code> with a - forward type of route, nat, or no forward at all). Moreover, the - virtual network the interface is connected to is required to have at least - inbound QoS set (<code>average</code> at least). Moreover, with - <code>floor</code> attribute users don't need to specify - <code>average</code>. However, <code>peak</code> and <code>burst</code> - attributes still require <code>average</code>. Currently, linux kernel - doesn't allow ingress qdiscs to have any classes therefore - <code>floor</code> can be applied only on <code>inbound</code> and not - <code>outbound</code>. <span class="since">Since 1.0.1</span> + and outgoing traffic can be shaped independently. + The <code>bandwidth</code> element and its child elements are described + in the <a href="formatnetwork.html#elementQoS">QoS</a> section of + the Network XML. </p>
<h5><a name="elementVlanTag">Setting VLAN tag (on supported network types only)</a></h5> diff --git a/docs/formatnetwork.html.in b/docs/formatnetwork.html.in index 1ca1bec..d4c390a 100644 --- a/docs/formatnetwork.html.in +++ b/docs/formatnetwork.html.in @@ -412,40 +412,81 @@
<p> The <code><bandwidth></code> element allows setting - quality of service for a particular network. - <span class="since">Since 0.9.4</span> The limits specified + quality of service for a particular network + (<span class="since">since 0.9.4</span>). For a <code>domain</code> + object, the limits specified are applied to the domain traffic.
I'm quite sure about the 'domain traffic'. The <bandwidth/> under domain limits the particular <interface/> that has <bandwidth/>. Having 'domain traffic' written here may sound like if the domain traffic was aggregated and then shaped (which is done in network not in domain). Maybe 'domain interface traffic'? Other than that, the patch looks good. Michal
<h5><a name="elementVlanTag">Setting VLAN tag (on supported network types only)</a></h5> diff --git a/docs/formatnetwork.html.in b/docs/formatnetwork.html.in index 1ca1bec..d4c390a 100644 --- a/docs/formatnetwork.html.in +++ b/docs/formatnetwork.html.in @@ -412,40 +412,81 @@
<p> The <code><bandwidth></code> element allows setting - quality of service for a particular network. - <span class="since">Since 0.9.4</span> The limits specified + quality of service for a particular network + (<span class="since">since 0.9.4</span>). For a <code>domain</code> + object, the limits specified are applied to the domain traffic.
I'm quite sure about the 'domain traffic'. The <bandwidth/> under domain limits the particular <interface/> that has <bandwidth/>. Having 'domain traffic' written here may sound like if the domain traffic was aggregated and then shaped (which is done in network not in domain). Maybe 'domain interface traffic'?
Right - I struggled with this one when trying to merge things. Using "domain interface traffic" does target a bit better. Of course the context is discussing network bandwidth, so I guess I just assumed - hah - that the reader would consider the domain traffic as the traffic for that interface. In the end, they both affect the same target interface; however, it wasn't quite clear which would take precedence. If the interface had one set of values on it, then the domain was defined with a different set, then what happens? Or vice versa - a domain with a set of values applied to an interface and then someone edits the network interface. It seems it would be the "last" set to be applied that would "win". Since this is an active topic for other recent patches, hopefully we can come up with a wording that will stick going forward. I've copied Laine on this response just to bring it to his attention and get his take as well... Tks, John
On 02/17/2014 05:45 PM, John Ferlan wrote:
<h5><a name="elementVlanTag">Setting VLAN tag (on supported network types only)</a></h5> diff --git a/docs/formatnetwork.html.in b/docs/formatnetwork.html.in index 1ca1bec..d4c390a 100644 --- a/docs/formatnetwork.html.in +++ b/docs/formatnetwork.html.in @@ -412,40 +412,81 @@
<p> The <code><bandwidth></code> element allows setting - quality of service for a particular network. - <span class="since">Since 0.9.4</span> The limits specified + quality of service for a particular network + (<span class="since">since 0.9.4</span>). For a <code>domain</code> + object, the limits specified are applied to the domain traffic.
I'm quite sure about the 'domain traffic'. The <bandwidth/> under domain limits the particular <interface/> that has <bandwidth/>. Having 'domain traffic' written here may sound like if the domain traffic was aggregated and then shaped (which is done in network not in domain). Maybe 'domain interface traffic'?
Right - I struggled with this one when trying to merge things. Using "domain interface traffic" does target a bit better. Of course the context is discussing network bandwidth, so I guess I just assumed - hah - that the reader would consider the domain traffic as the traffic for that interface.
In the end, they both affect the same target interface; however, it wasn't quite clear which would take precedence. If the interface had one set of values on it, then the domain was defined with a different set, then what happens?
That is a non-question, because there is no <bandwidth> subelement of a <domain>! The <bandwidth> element can appear in 3 places: 1) as a subelement of a domain's <interface>. In this case, the bandwidth applies to that one interface of that domain. 2) as a subelement of a <portgroup> in a <network>. In this case, if a domain's <interface> has a "portgroup='xxx'" attribute in its <source> element: <interface type='network'> <source network='testnet' portgroup='admin'/> ... *and* if the <interface itself has no <bandwidth> element, then the <bandwidth> element of the portgroup is applied to that interface. 3) as a subelement of a <network>. In this case, the bandwidth specified is intended to be to total aggregate bandwidth of all guest interfaces attached to that network. So, as you can see, there is no concept of "the bandwidth of a domain". The places where there may be a question of precendence: 1) if an <interface> has a <bandwidth> *and* it belongs to a portgroup that has a <bandwidth>. In this case, we decided that the bandwidth under the individual <interface> would take precedence, since at the time anyone who had permission to edit the domain (and its <interface>) also had permission to edit the network (and its <portgroup>); so allowing the specific config for one interface to override the general config for many interfaces was more useful. 2) if a <network> has a bandwidth defined for the aggregate of all interfaces attached to that network, and one interface has bandwidth defined that is higher than the aggregate for the entire network. In this case, the aggregate bandwidth for the network takes precedence, not because of anything we do, but just because the two choke points are independent of each other (the <interface> bandwidth control happens on the interface's tap device, and the <network> bandwidth control happens on the interface part of the bridge device created for that network). Did you maybe mean to say If the interface had one set of values on it, then the *NETWORK* was defined with a different set, then what happens? ? If so, then I believe (2) above explains in (rather obtusely, of course, but I hope it's possible to decipher it)
Or vice versa - a domain with a set of values applied to an interface and then someone edits the network interface. It seems it would be the "last" set to be applied that would "win".
What is "interface" vs. "network interface"? I can't tell if you're talking about the same thing with two slightly different names, or incorrectly naming something in the <network> configuration. But if you're talking about interface bandwidth vs. network aggregate bandwidth, the way that it works is that both sets of rules are installed, and the traffic for that guest's interface is shaped twice - once as it goes in/out of the guest interface's tap device, then the same traffic is shaped again (in aggregate with all other traffic for the network) as it goes in/out of the interface that is a part of the network's bridge device.
Since this is an active topic for other recent patches, hopefully we can come up with a wording that will stick going forward. I've copied Laine on this response just to bring it to his attention and get his take as well...
Although I understand the desire to eliminate nearly duplicated wording, I'm a bit concerned about trying to combine two things that are almost but not entirely the same, due to the potential of the combined information being wrong in one of both cases, or at least becoming unnecessarily confusing due to extra "exceptions" that are needed to explain the differences. In spite of that, I'm fine with it being combined - if people complain, we can always split it up again. I can never really form a good opinion about documentation without seeing it live on the web page and hearing how others interpret it, and my burn-out level on documentation has lately led to my opinion either being "yeah, that's okay", or "no, I don't like that, but I don't know how to / don't have time to fix it" :-P (The recent change to the information for <bandwidth> in the network documentation was a side effect of me making a small change to validation of network xml to disallow specifying <bandwidth> on network types where we can't support setting aggregate bandwidth for the entire network. While I was doing that, I noticed that the documentation about <bandwidth> in formatnetwork.html.in was obtuse/incorrect, so I made some changes to fix those things; I didn't think to look at the documentation for <bandwidth> in the domain (<interface>) XML docs, because I hadn't changed anything there.)
On 02/18/2014 05:58 AM, Laine Stump wrote:
On 02/17/2014 05:45 PM, John Ferlan wrote:
<h5><a name="elementVlanTag">Setting VLAN tag (on supported network types only)</a></h5> diff --git a/docs/formatnetwork.html.in b/docs/formatnetwork.html.in index 1ca1bec..d4c390a 100644 --- a/docs/formatnetwork.html.in +++ b/docs/formatnetwork.html.in @@ -412,40 +412,81 @@
<p> The <code><bandwidth></code> element allows setting - quality of service for a particular network. - <span class="since">Since 0.9.4</span> The limits specified + quality of service for a particular network + (<span class="since">since 0.9.4</span>). For a <code>domain</code> + object, the limits specified are applied to the domain traffic.
I'm quite sure about the 'domain traffic'. The <bandwidth/> under domain limits the particular <interface/> that has <bandwidth/>. Having 'domain traffic' written here may sound like if the domain traffic was aggregated and then shaped (which is done in network not in domain). Maybe 'domain interface traffic'?
Right - I struggled with this one when trying to merge things. Using "domain interface traffic" does target a bit better. Of course the context is discussing network bandwidth, so I guess I just assumed - hah - that the reader would consider the domain traffic as the traffic for that interface.
In the end, they both affect the same target interface; however, it wasn't quite clear which would take precedence. If the interface had one set of values on it, then the domain was defined with a different set, then what happens?
That is a non-question, because there is no <bandwidth> subelement of a <domain>! The <bandwidth> element can appear in 3 places:
1) as a subelement of a domain's <interface>. In this case, the bandwidth applies to that one interface of that domain.
2) as a subelement of a <portgroup> in a <network>. In this case, if a domain's <interface> has a "portgroup='xxx'" attribute in its <source> element:
<interface type='network'> <source network='testnet' portgroup='admin'/> ...
*and* if the <interface itself has no <bandwidth> element, then the <bandwidth> element of the portgroup is applied to that interface.
3) as a subelement of a <network>. In this case, the bandwidth specified is intended to be to total aggregate bandwidth of all guest interfaces attached to that network.
So, as you can see, there is no concept of "the bandwidth of a domain".
Semantics - I suppose. The "whole" network vs. the "slice" for the domain. Trying to understand while flipping between two pages and using "improper" or "inexact" words. I guess what I meant was the bandwidth assigned to the interface of a domain vs. the bandwidth assigned to the whole network. In any case, you've answered my questions below...
The places where there may be a question of precendence:
1) if an <interface> has a <bandwidth> *and* it belongs to a portgroup that has a <bandwidth>. In this case, we decided that the bandwidth under the individual <interface> would take precedence, since at the time anyone who had permission to edit the domain (and its <interface>) also had permission to edit the network (and its <portgroup>); so allowing the specific config for one interface to override the general config for many interfaces was more useful.
2) if a <network> has a bandwidth defined for the aggregate of all interfaces attached to that network, and one interface has bandwidth defined that is higher than the aggregate for the entire network. In this case, the aggregate bandwidth for the network takes precedence, not because of anything we do, but just because the two choke points are independent of each other (the <interface> bandwidth control happens on the interface's tap device, and the <network> bandwidth control happens on the interface part of the bridge device created for that network).
Did you maybe mean to say
If the interface had one set of values on it, then the *NETWORK* was defined with a different set, then what happens?
? If so, then I believe (2) above explains in (rather obtusely, of course, but I hope it's possible to decipher it)
Your (2) is closer to what I was going for... and yes it makes sense. The whole portgroup discussion (1) adds complexity to things, but it is described in the formatnetwork page, although it was in a "different" paragraph which is why I ignored it at first. I've since moved it (see below)
Or vice versa - a domain with a set of values applied to an interface and then someone edits the network interface. It seems it would be the "last" set to be applied that would "win".
What is "interface" vs. "network interface"? I can't tell if you're talking about the same thing with two slightly different names, or incorrectly naming something in the <network> configuration.
Again some semantics, but it's becoming more clear now - thanks. The reference to "network interface" was the "<network>" object as opposed to "an interface" which was the "<domain><interface>" object (again I may not be as semantically correct as you expect, but hopefully you see what I was intending).
But if you're talking about interface bandwidth vs. network aggregate bandwidth, the way that it works is that both sets of rules are installed, and the traffic for that guest's interface is shaped twice - once as it goes in/out of the guest interface's tap device, then the same traffic is shaped again (in aggregate with all other traffic for the network) as it goes in/out of the interface that is a part of the network's bridge device.
A discussion which I think would be good to add to formatnetwork...
Since this is an active topic for other recent patches, hopefully we can come up with a wording that will stick going forward. I've copied Laine on this response just to bring it to his attention and get his take as well...
Although I understand the desire to eliminate nearly duplicated wording, I'm a bit concerned about trying to combine two things that are almost but not entirely the same, due to the potential of the combined information being wrong in one of both cases, or at least becoming unnecessarily confusing due to extra "exceptions" that are needed to explain the differences. In spite of that, I'm fine with it being combined - if people complain, we can always split it up again.
Understood; however, considering your last paragraph regarding missing or not thinking about formatdomain is just cause enough to try and put the discussion in one place. Keeping things in two places usually results in one being forgotten during editing. I used cscope to find relevant bandwidth discussions...
I can never really form a good opinion about documentation without seeing it live on the web page and hearing how others interpret it, and my burn-out level on documentation has lately led to my opinion either being "yeah, that's okay", or "no, I don't like that, but I don't know how to / don't have time to fix it" :-P
(The recent change to the information for <bandwidth> in the network documentation was a side effect of me making a small change to validation of network xml to disallow specifying <bandwidth> on network types where we can't support setting aggregate bandwidth for the entire network. While I was doing that, I noticed that the documentation about <bandwidth> in formatnetwork.html.in was obtuse/incorrect, so I made some changes to fix those things; I didn't think to look at the documentation for <bandwidth> in the domain (<interface>) XML docs, because I hadn't changed anything there.)
Because I know it's easier to read without all the git +/-, here's what I have now in the QoS section on the formatnetwork (hopefully I've captured everything correctly): The <bandwidth> element allows setting quality of service for a particular network (since 0.9.4). Setting bandwidth for a network is supported only for networks with a <forward> mode of route, nat, or no mode at all (i.e. an "isolated" network). Setting bandwidth is not supported for forward modes of bridge, passthrough, private, or hostdev. Attempts to do this will lead to a failure to define the network or to create a transient network. The <bandwidth> element can only be a subelement of a domain's <interface>, a subelement of a <network>, or a subelement of a <portgroup> in a <network>. As a subelement of a domain's <interface>, the bandwidth only applies to that one interface of the domain. As a subelement of a <network>, the bandwidth is a total aggregate bandwidth to/from all guest interfaces attached to that network, not to each guest interface individually. If a domain's <interface> has <bandwidth> element values higher than the aggregate for the entire network, then the aggregate bandwidth for the <network> takes precedence. This is because the two choke points are independent of each other where the domain's <interface> bandwidth control is applied on the interface's tap device, while the <network> bandwidth control is applied on the interface part of the bridge device created for that network. As a subelement of a <portgroup> in a <network>, if a domain's <interface> has a portgroup attribute in its <source> element and if the <interface> itself has no <bandwidth> element, then the <bandwidth> element of the portgroup will be applied individually to each guest interface defined to be a member of that portgroup. Any <bandwidth> element in the domain's <interface> definition will override the setting in the portgroup (since 1.0.1). Incoming and outgoing traffic can be shaped independently. The bandwidth element can have at most one inbound and at most one outbound child element. Leaving either of these children elements out results in no QoS applied for that traffic direction. So, when you want to shape only incoming traffic, use inbound only, and vice versa. Each of these elements have one mandatory attribute - average (or floor as described below). The attributes are as follows, where accepted values for each attribute is an integer number. average Specifies the desired average bit rate for the interface being shaped (in kilobytes/second). peak Optional attribute which specifies the maximum rate at which the bridge can send data (in kilobytes/second). Note the limitation of implementation: this attribute in the outbound element is ignored (as Linux ingress filters don't know it yet). burst Optional attribute which specifies the amount of kilobytes that can be transmitted in a single burst at peak speed. floor Optional attribute available only for the inbound element. This attribute guarantees minimal throughput for shaped interfaces. This, however, requires that all traffic goes through one point where QoS decisions can take place, hence why this attribute works only for virtual networks for now (that is <interface type='network'/> with a forward type of route, nat, or no forward at all). Moreover, the virtual network the interface is connected to is required to have at least inbound QoS set (average at least). If using the floor attribute users don't need to specify average. However, peak and burst attributes still require average. Currently, the Linux kernel doesn't allow ingress qdiscs to have any classes therefore floor can be applied only on inbound and not outbound. Attributes average, peak, and burst are available since 0.9.4, while the floor attribute is available since 1.0.1.
On 02/18/2014 07:16 PM, John Ferlan wrote:
Because I know it's easier to read without all the git +/-, here's what I have now in the QoS section on the formatnetwork (hopefully I've captured everything correctly):
This all looks fine. ACK.
The <bandwidth> element allows setting quality of service for a particular network (since 0.9.4). Setting bandwidth for a network is supported only for networks with a <forward> mode of route, nat, or no mode at all (i.e. an "isolated" network). Setting bandwidth is not supported for forward modes of bridge, passthrough, private, or hostdev. Attempts to do this will lead to a failure to define the network or to create a transient network.
The <bandwidth> element can only be a subelement of a domain's <interface>, a subelement of a <network>, or a subelement of a <portgroup> in a <network>.
As a subelement of a domain's <interface>, the bandwidth only applies to that one interface of the domain. As a subelement of a <network>, the bandwidth is a total aggregate bandwidth to/from all guest interfaces attached to that network, not to each guest interface individually. If a domain's <interface> has <bandwidth> element values higher than the aggregate for the entire network, then the aggregate bandwidth for the <network> takes precedence. This is because the two choke points are independent of each other where the domain's <interface> bandwidth control is applied on the interface's tap device, while the <network> bandwidth control is applied on the interface part of the bridge device created for that network.
As a subelement of a <portgroup> in a <network>, if a domain's <interface> has a portgroup attribute in its <source> element and if the <interface> itself has no <bandwidth> element, then the <bandwidth> element of the portgroup will be applied individually to each guest interface defined to be a member of that portgroup. Any <bandwidth> element in the domain's <interface> definition will override the setting in the portgroup (since 1.0.1).
Incoming and outgoing traffic can be shaped independently. The bandwidth element can have at most one inbound and at most one outbound child element. Leaving either of these children elements out results in no QoS applied for that traffic direction. So, when you want to shape only incoming traffic, use inbound only, and vice versa. Each of these elements have one mandatory attribute - average (or floor as described below). The attributes are as follows, where accepted values for each attribute is an integer number.
average Specifies the desired average bit rate for the interface being shaped (in kilobytes/second).
peak Optional attribute which specifies the maximum rate at which the bridge can send data (in kilobytes/second). Note the limitation of implementation: this attribute in the outbound element is ignored (as Linux ingress filters don't know it yet).
burst Optional attribute which specifies the amount of kilobytes that can be transmitted in a single burst at peak speed.
floor Optional attribute available only for the inbound element. This attribute guarantees minimal throughput for shaped interfaces. This, however, requires that all traffic goes through one point where QoS decisions can take place, hence why this attribute works only for virtual networks for now (that is <interface type='network'/> with a forward type of route, nat, or no forward at all). Moreover, the virtual network the interface is connected to is required to have at least inbound QoS set (average at least). If using the floor attribute users don't need to specify average. However, peak and burst attributes still require average. Currently, the Linux kernel doesn't allow ingress qdiscs to have any classes therefore floor can be applied only on inbound and not outbound.
Attributes average, peak, and burst are available since 0.9.4, while the floor attribute is available since 1.0.1.
participants (3)
-
John Ferlan -
Laine Stump -
Michal Privoznik