does the 'bandwidth' work?
Hi guys. I have a network like so: <network> <name>10_1_1</name> <uuid>864fb78e-0fb0-4c32-bb47-0e5ac68d9491</uuid> <forward mode='bridge'/> <bridge name='nm-bridge1011'/> <bandwidth> <inbound average='610351' peak='620000'/> <outbound average='610351' peak='620000'/> </bandwidth> </network> I'd expect this to limit network to 10Gbs - since this is in kibibytes - I assume these two add to make the max? But when I _iperf3_ in guests then I get speeds near host's native, which are much higher than expected limit. Am I missing something - no <bandwidth> in guests - or there actually might be an issue here? I'm on Centos10 with libvirt-12.8.0-1.el10 thanks, L.
On 10/8/26 10:33, lejeczek via Users wrote:
Hi guys.
I have a network like so:
<network> <name>10_1_1</name> <uuid>864fb78e-0fb0-4c32-bb47-0e5ac68d9491</uuid> <forward mode='bridge'/> <bridge name='nm-bridge1011'/> <bandwidth> <inbound average='610351' peak='620000'/> <outbound average='610351' peak='620000'/> </bandwidth> </network>
I'd expect this to limit network to 10Gbs - since this is in kibibytes - I assume these two add to make the max? But when I _iperf3_ in guests then I get speeds near host's native, which are much higher than expected limit. Am I missing something - no <bandwidth> in guests - or there actually might be an issue here? I'm on Centos10 with libvirt-12.8.0-1.el10
Be aware that this config sets QoS on nm-bridge1011 bridge (which you can verify by running `tc -s -d class show dev nm-bridge1011`). But the thing is, when the virtualization hosts wants to send a packet to a guest it bypasses bridge and talks to vnetX directly. Try running iperf from a different host. Michal
Apologies, when I said "in guests" I meant only guests I was iper'fing. To clarify: guests on different but virtually identical hosts - iperf3 between such guests. What should a "typical" _tc_ look like with libvirt doing bandwidth management? Seems that libvirt does QoS on the hosts - this is with a single net bandwidth'ed: -> $ tc -s -d class show dev nm-bridge1011 class htb 1:1 root rate 4882Mbit ceil 4960Mbit linklayer ethernet burst 1601b/1mpu 0b cburst 1626b/1mpu 0b level 7 Sent 9559833379 bytes 10652255 pkt (dropped 0, overlimits 594941 requeues 0) backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 39 ctokens: 39 class htb 1:2 parent 1:1 leaf 2: prio 0 quantum 52083 rate 4882Mbit ceil 4960Mbit linklayer ethernet burst 1601b/1mpu 0b cburst 1626b/1mpu 0b level 0 Sent 9559833379 bytes 10652255 pkt (dropped 0, overlimits 641156 requeues 0) backlog 0b 0p requeues 0 lended: 10399228 borrowed: 0 giants: 0 tokens: 39 ctokens: 39 and yet... What might be not/happening here? thanks, L.
On 10/8/26 14:57, lejeczek via Users wrote:
Apologies, when I said "in guests" I meant only guests I was iper'fing. To clarify: guests on different but virtually identical hosts - iperf3 between such guests.
What should a "typical" _tc_ look like with libvirt doing bandwidth management? Seems that libvirt does QoS on the hosts - this is with a single net bandwidth'ed:
-> $ tc -s -d class show dev nm-bridge1011 class htb 1:1 root rate 4882Mbit ceil 4960Mbit linklayer ethernet burst
Here you can see that the average is set correctly (4882Mbit corresponds to the 'average' from your previous e-mail, and 4960 corresponds to the 'peak').
1601b/1mpu 0b cburst 1626b/1mpu 0b level 7 Sent 9559833379 bytes 10652255 pkt (dropped 0, overlimits 594941 requeues 0)
And here you can see there's traffic shaping happening. There was ~8.9GB sent and ~0.59M packets were "borrowed" from the 'peak'.
backlog 0b 0p requeues 0 lended: 0 borrowed: 0 giants: 0 tokens: 39 ctokens: 39
class htb 1:2 parent 1:1 leaf 2: prio 0 quantum 52083 rate 4882Mbit ceil 4960Mbit linklayer ethernet burst 1601b/1mpu 0b cburst 1626b/1mpu 0b level 0 Sent 9559833379 bytes 10652255 pkt (dropped 0, overlimits 641156 requeues 0) backlog 0b 0p requeues 0 lended: 10399228 borrowed: 0 giants: 0 tokens: 39 ctokens: 39
This is then a queuing class where the default traffic goes. Long story short, <interface>-s from domain XML that have 'floor' set would create another class here (1:3, 1:4, ...) and their traffic would be shaped through that class. Everything else, i.e. <interface>-s that have no 'floor' or no <bandwidth> are shaped through this 1:2 class. Hence the same setting as 1:1. If you want to learn more about how libvirt sets up QoS there's a huge comment with a picture inside our sources [1].
and yet... What might be not/happening here? thanks, L.
This is weird. I mean, if two VMs are sending packets to each other then their traffic is routed through the bridge and thus it is shaped. It's only when the host talks to a VM the bridge is bypassed (this is in fact a "feature" of linux bridge, which is not L2 bridge really). Anyway - so you're saying that you have two VMs connected to the same bridge and their traffic between each other is not shaped? Where exactly is iperf running and where is it connecting to? Michal 1: https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virnetdevbandwidth...
Oh yes, this can be quite confusing to an admin, it is to me. Seems that using network bandwidth - as opposed to limits per guest - limits bandwidth on host & thus between the hosts while letting guests keep... full speed? if this is how it is... supposed to be - this might fool many admins, it did fool me. (this should be exclaimed in docs, even better in a man page - if @devel read this) Is there a way to achieve this simple goal I'm trying to achieve - without going into each guest definition - single entry-point for bandwidth limit for guests using given network(& not cripple host "outside" networking)? thanks, L.
participants (2)
-
lejeczek -
Michal Prívozník