[libvirt-users] Libvirt "tc ingress qdisc" automatically removed by ovs vlan tag setting, how?
With outbound QoS setting in Libvirt XML, libvirt will add a tc ingress qdisc for traffic shaping. Then if you set VLAN tag to that tap device, this qdisc will automatically gone by no reason. Could anyone shed some lights where should I look into? I'm really confused and got no clue here. Thanks! Steps to reproduce -- # virsh start instance-name # virsh dumpxml instance-name ... <interface type='bridge'> <mac address='fa:16:3e:b9:8f:2a'/> <source bridge='br-int'/> <virtualport type='openvswitch'> <parameters interfaceid='0a2b02ca-4824-4bda-baa9-05fff7a3146d'/> </virtualport> <target dev='tap0a2b02ca-48'/> <model type='virtio'/> <bandwidth> <outbound average='256' peak='512' burst='512'/> </bandwidth> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> ... # tc qdisc ls dev tap0a2b02ca-48 qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 qdisc sfq 2: parent 1:1 limit 127p quantum 1514b perturb 10sec qdisc ingress ffff: parent ffff:fff1 ---------------- (<-- already showed up) # ovs-vsctl set port tap0a2b02ca-48 tag=1 # tc qdisc ls dev tap0a2b02ca-48 qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 qdisc sfq 2: parent 1:1 limit 127p quantum 1514b perturb 10sec (<-- ingress qdisc is gone) After that, if add this qdisc manually and re-set vlan tag, problem no longer exists. But it can always be reproduced after "virsh destroy / start" cycle. # tc qdisc add dev tap0a2b02ca-48 handle ffff: ingress # ovs-vsctl clear port tap0a2b02ca-48 tag # ovs-vsctl set port tap0a2b02ca-48 tag=1 # tc qdisc ls dev tap0a2b02ca-48 qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 qdisc sfq 2: parent 1:1 limit 127p quantum 1514b perturb 10sec qdisc ingress ffff: parent ffff:fff1 ---------------- (<-- still exists) Other information -- # ovs-vsctl --version ovs-vsctl (Open vSwitch) 1.9.2 Compiled May 27 2013 14:19:16 # uname -r 2.6.32-358.111.1.openstack.el6.x86_64 # virsh --version (This is a version I built from libvirt upstream origin/v0.9.11-stable branch) 0.9.11.9 -- Qiu Yu
After some digging in openvswitch code. My wild guess is that vlan tag reconfiguring triggered iface_configure_qos (vswitchd/bridge.c), which in turn called netdev_set_policing to reset ingress policing rate. Although there's no ingress_policing_rate set in my case, existing ingress qdisc still remove by default. Could some openvswitch guy help to confirm and suggest how to fix or workaround? It seems to be a bug of incompatible with existing ingress qdisc settings. Thanks, -- Qiu Yu On Wed, Jul 17, 2013 at 8:09 PM, Qiu Yu <unicell@gmail.com> wrote:
With outbound QoS setting in Libvirt XML, libvirt will add a tc ingress qdisc for traffic shaping. Then if you set VLAN tag to that tap device, this qdisc will automatically gone by no reason.
Could anyone shed some lights where should I look into? I'm really confused and got no clue here. Thanks!
Steps to reproduce -- # virsh start instance-name # virsh dumpxml instance-name
... <interface type='bridge'> <mac address='fa:16:3e:b9:8f:2a'/> <source bridge='br-int'/> <virtualport type='openvswitch'> <parameters interfaceid='0a2b02ca-4824-4bda-baa9-05fff7a3146d'/> </virtualport> <target dev='tap0a2b02ca-48'/> <model type='virtio'/> <bandwidth> <outbound average='256' peak='512' burst='512'/> </bandwidth> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> ...
# tc qdisc ls dev tap0a2b02ca-48 qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 qdisc sfq 2: parent 1:1 limit 127p quantum 1514b perturb 10sec qdisc ingress ffff: parent ffff:fff1 ---------------- (<-- already showed up)
# ovs-vsctl set port tap0a2b02ca-48 tag=1 # tc qdisc ls dev tap0a2b02ca-48 qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 qdisc sfq 2: parent 1:1 limit 127p quantum 1514b perturb 10sec (<-- ingress qdisc is gone)
After that, if add this qdisc manually and re-set vlan tag, problem no longer exists. But it can always be reproduced after "virsh destroy / start" cycle.
# tc qdisc add dev tap0a2b02ca-48 handle ffff: ingress # ovs-vsctl clear port tap0a2b02ca-48 tag # ovs-vsctl set port tap0a2b02ca-48 tag=1 # tc qdisc ls dev tap0a2b02ca-48 qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 qdisc sfq 2: parent 1:1 limit 127p quantum 1514b perturb 10sec qdisc ingress ffff: parent ffff:fff1 ---------------- (<-- still exists)
Other information -- # ovs-vsctl --version ovs-vsctl (Open vSwitch) 1.9.2 Compiled May 27 2013 14:19:16 # uname -r 2.6.32-358.111.1.openstack.el6.x86_64 # virsh --version (This is a version I built from libvirt upstream origin/v0.9.11-stable branch) 0.9.11.9
-- Qiu Yu
On Wed, Jul 17, 2013 at 6:06 AM, Qiu Yu <unicell@gmail.com> wrote:
After some digging in openvswitch code. My wild guess is that vlan tag reconfiguring triggered iface_configure_qos (vswitchd/bridge.c), which in turn called netdev_set_policing to reset ingress policing rate. Although there's no ingress_policing_rate set in my case, existing ingress qdisc still remove by default.
The OVS database in general specifies state, not actions. That is, if you set no ingress_policing_rate, then OVS takes that to mean that it should turn off ingress policing, so it does. I'm surprised that you actually find ingress policing useful. Our experience is that it doesn't work well, regardless of how you configure it.
On Thu, Jul 18, 2013 at 12:15 AM, Ben Pfaff <blp@nicira.com> wrote:
On Wed, Jul 17, 2013 at 6:06 AM, Qiu Yu <unicell@gmail.com> wrote:
After some digging in openvswitch code. My wild guess is that vlan tag reconfiguring triggered iface_configure_qos (vswitchd/bridge.c), which in turn called netdev_set_policing to reset ingress policing rate. Although there's no ingress_policing_rate set in my case, existing ingress qdisc still remove by default.
The OVS database in general specifies state, not actions. That is, if you set no ingress_policing_rate, then OVS takes that to mean that it should turn off ingress policing, so it does.
I see. So ingress policing managed outside of OVS, which is libvirt in my case, is overridden (no ingress_policing_rate by default, means turned off) by OVS. My question, however, is there any workaround, to preserve or inherit the ingress policing managed outside of OVS after vlan tagging the device?
I'm surprised that you actually find ingress policing useful. Our experience is that it doesn't work well, regardless of how you configure it.
Well, I'm not aware of other software solution to achieve ingress throttling purpose. :( For me, it is better than nothing. Any alternative solution to share? :) -- Qiu Yu
On Thu, Jul 18, 2013 at 12:30:19AM +0800, Qiu Yu wrote:
On Thu, Jul 18, 2013 at 12:15 AM, Ben Pfaff <blp@nicira.com> wrote:
On Wed, Jul 17, 2013 at 6:06 AM, Qiu Yu <unicell@gmail.com> wrote:
After some digging in openvswitch code. My wild guess is that vlan tag reconfiguring triggered iface_configure_qos (vswitchd/bridge.c), which in turn called netdev_set_policing to reset ingress policing rate. Although there's no ingress_policing_rate set in my case, existing ingress qdisc still remove by default.
The OVS database in general specifies state, not actions. That is, if you set no ingress_policing_rate, then OVS takes that to mean that it should turn off ingress policing, so it does.
I see. So ingress policing managed outside of OVS, which is libvirt in my case, is overridden (no ingress_policing_rate by default, means turned off) by OVS.
My question, however, is there any workaround, to preserve or inherit the ingress policing managed outside of OVS after vlan tagging the device?
OVS doesn't have one now. We'd take a patch to introduce one. "VLAN tagging the device" is a bit of a misunderstanding. This behavior will take place on any change to Open vSwitch configuration. VLAN tagging is just the one example that you have encountered.
On 17.07.2013 18:30, Qiu Yu wrote:
On Thu, Jul 18, 2013 at 12:15 AM, Ben Pfaff <blp@nicira.com> wrote:
On Wed, Jul 17, 2013 at 6:06 AM, Qiu Yu <unicell@gmail.com> wrote:
After some digging in openvswitch code. My wild guess is that vlan tag reconfiguring triggered iface_configure_qos (vswitchd/bridge.c), which in turn called netdev_set_policing to reset ingress policing rate. Although there's no ingress_policing_rate set in my case, existing ingress qdisc still remove by default.
The OVS database in general specifies state, not actions. That is, if you set no ingress_policing_rate, then OVS takes that to mean that it should turn off ingress policing, so it does.
I see. So ingress policing managed outside of OVS, which is libvirt in my case, is overridden (no ingress_policing_rate by default, means turned off) by OVS.
My question, however, is there any workaround, to preserve or inherit the ingress policing managed outside of OVS after vlan tagging the device?
I'm surprised that you actually find ingress policing useful. Our experience is that it doesn't work well, regardless of how you configure it.
Well, I'm not aware of other software solution to achieve ingress throttling purpose. :( For me, it is better than nothing. Any alternative solution to share? :)
There's one. It's called IFB (Intermediate Functional Block). This is basically a pair of interfaces that act like bidirectional pipe. What is sent at one end, appears on the other one and vice versa. Advantage is, you can do actual shaping, not just policing. As IFB name says, you can insert this element between vNIC from domain and the bridge, and effectively turn incoming traffic to outgoing. And since linux kernel doesn't allow us to shape incoming, just outgoing traffic, you can easily use shaping with IFB. There are, however, couple of reasons why not to go down this path. Firstly, much higher overhead due to packets going longer paths through linux networking internals. Secondly, IFB devices cannot be created on the fly. At the time of IFB module load, one can specify how many IFB devices shall be created, but this cannot be changed thereafter. These are the reasons I've decided to go with 'just' policing. Michal The difference between policing and actual shaping is: while in shaping packets are placed into a buffer and delivered delayed, in policing a packet which is above the limit is just dropped. IOW, policing just cuts off the peaks. TCP can deal with both, though.
participants (3)
-
Ben Pfaff -
Michal Privoznik -
Qiu Yu