… Hello. I would like to start a discussion about network SLA mechs in libvirt. Right now, as I understand, libvirt is managing traffic controls, e.g. bandwitdh limitation, calling external routines (ip/tc). This approach has its own drawbacks, including the need to parse text output of external commands and also some complexity with the current state identification — the queueing discipline can be changed on the interface in runtime outside of libvirt. But I would like to propose another approach — not dropping the previous one, just as an option, maybe. Not so long time ago I started a project [1] that works with netlink directly and can provide standalone daemon, that manages interface properties, including queueing disciplines. Working with netlink events, it is in permanent sync with the current interface statuses w/o any polling. So maybe it would make sense to use this daemon — e.g. with JSON or XML-RPC over SSL/TLS, or with any other RPC that's preferred by libvirt. That approach can provide: * more high-level API, that will keep libvirt off the need to compute rule and filter handlers, hierarchy and so on — we would be able to implement more complex traffic control schemes in more easy way * generally speaking, an RPC API is more easy to use, than the execution of external binary file and parsing text output, and `tc` is not the easiest command to automate. If it sounds reasonably enough, we can discuss the overall scheme, communication and API details that would be more comfortable to libvirt and so on. So, any comments? [1] — https://github.com/svinota/pyroute2 -- Peter V. Saveliev
On 17.05.2013 13:07, Peter V. Saveliev wrote:
…
Hello.
I would like to start a discussion about network SLA mechs in libvirt.
Right now, as I understand, libvirt is managing traffic controls, e.g. bandwitdh limitation, calling external routines (ip/tc). This approach has its own drawbacks, including the need to parse text output of external commands and also some complexity with the current state identification — the queueing discipline can be changed on the interface in runtime outside of libvirt.
But I would like to propose another approach — not dropping the previous one, just as an option, maybe. Not so long time ago I started a project [1] that works with netlink directly and can provide standalone daemon, that manages interface properties, including queueing disciplines. Working with netlink events, it is in permanent sync with the current interface statuses w/o any polling. So maybe it would make sense to use this daemon — e.g. with JSON or XML-RPC over SSL/TLS, or with any other RPC that's preferred by libvirt.
That approach can provide:
* more high-level API, that will keep libvirt off the need to compute rule and filter handlers, hierarchy and so on — we would be able to implement more complex traffic control schemes in more easy way
* generally speaking, an RPC API is more easy to use, than the execution of external binary file and parsing text output, and `tc` is not the easiest command to automate.
In fact, there's currently no 'tc' output parsing at all. None is needed. When setting QoS, libvirt just flushes all previous settings and insert its own. I don't think libvirt can guarantee things like minimal throughput if we let other applications to interfere with libvirt QoS settings. So our reason should be slightly different: offloading QoS setting to a external application which in turn allows us to create even more complicated QoS tree. This would be a reasonable trade off.
If it sounds reasonably enough, we can discuss the overall scheme, communication and API details that would be more comfortable to libvirt and so on.
Regarding communication with your daemon we have 2 options: a) A monitor to which libvirt would connect b) A C library containing APIs that libvirt would call (internally, they will just hide a) and wrap it into a C function anyway). Since your application is pure python, going with b) would be like learning a tux to fly :) Hence I vote for a). Regarding protocol format, I prefer JSON as libvirt already has capability of parsing it and creating new messages. The other option is XML. Currently, the 'tc' is used to manage all three QoS objects: qdisc, class and filter. Take look at [2] to see the most complicated QoS tree that libvirt creates. Honestly, I am not sure how to catch that into an API calls. Michal
So, any comments?
2: http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/util/virnetdevbandwidth.c...
On Fri, May 17, 2013 at 01:07:02PM +0200, Peter V. Saveliev wrote:
…
Hello.
I would like to start a discussion about network SLA mechs in libvirt.
Right now, as I understand, libvirt is managing traffic controls, e.g. bandwitdh limitation, calling external routines (ip/tc). This approach has its own drawbacks, including the need to parse text output of external commands and also some complexity with the current state identification — the queueing discipline can be changed on the interface in runtime outside of libvirt.
If an administrator is changing stuff managed by libvirt behind its back, all bets are off & they are on their own when things break. The same as if you go making sending commands to the QEMU monitor directly, or do any number of other things. So the traffic shaping code really isn't unique in this respect, and doesn't need special handling to deal with that.
But I would like to propose another approach — not dropping the previous one, just as an option, maybe. Not so long time ago I started a project [1] that works with netlink directly and can provide standalone daemon, that manages interface properties, including queueing disciplines. Working with netlink events, it is in permanent sync with the current interface statuses w/o any polling. So maybe it would make sense to use this daemon — e.g. with JSON or XML-RPC over SSL/TLS, or with any other RPC that's preferred by libvirt.
That approach can provide:
* more high-level API, that will keep libvirt off the need to compute rule and filter handlers, hierarchy and so on — we would be able to implement more complex traffic control schemes in more easy way
* generally speaking, an RPC API is more easy to use, than the execution of external binary file and parsing text output, and `tc` is not the easiest command to automate.
Actually the 'tc' command is pretty trivial to use & we don't need to parse text output from it at all.
If it sounds reasonably enough, we can discuss the overall scheme, communication and API details that would be more comfortable to libvirt and so on.
The idea of handing off this to an external python daemon isn't really very appealing to me, because I don't really like libvirt core functionality depending on anything python related. Really libvirt should just talk network directly rather than invoking the 'tc' command, or there could be a low level C library that could be used instead of 'tc'. I don't think we need to hand this off to a daemon at all. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 05/17/2013 07:06 PM, Daniel P. Berrange wrote: <skip />
The idea of handing off this to an external python daemon isn't really very appealing to me, because I don't really like libvirt core functionality depending on anything python related.
Really libvirt should just talk network directly rather than invoking the 'tc' command, or there could be a low level C library that could be used instead of 'tc'. I don't think we need to hand this off to a daemon at all.
Does it matter, in what the language is written the external program, when it follows some standard API? Having once C implementation, that meets the requirements («no python»), one can always replace python implementation. Since there can be more appliances for such traffic controlling application, than only libvirt, standard API will make it possible to reuse it in other solutions — be it python implementation, C or even erlang. Python was choosen 'cause of one simple reason: it allows to implement netlink message encoder/decoder a way much easier than C, and since the netlink control message communication doesn't require high throughput, python's performance more than enough. Simple structure handling in python allows in this case really fast time-to-market features delivery, sometimes it makes sense. Beside of that, some high-level virtualization solutions uses python anyway, so mostly this application will not even create unique package dependencies, using only python stdlib and no tricks like ctypes and so on. But as I said, the language in this case really doesn't mean so much, having standard RPC API. C — ok, maybe someone will implement it in C also. The key issue is to have the API standard.
On Fri, May 17, 2013 at 08:04:30PM +0200, Peter V. Saveliev wrote:
On 05/17/2013 07:06 PM, Daniel P. Berrange wrote:
<skip />
The idea of handing off this to an external python daemon isn't really very appealing to me, because I don't really like libvirt core functionality depending on anything python related.
Really libvirt should just talk network directly rather than invoking the 'tc' command, or there could be a low level C library that could be used instead of 'tc'. I don't think we need to hand this off to a daemon at all.
Does it matter, in what the language is written the external program, when it follows some standard API? Having once C implementation, that meets the requirements («no python»), one can always replace python implementation. Since there can be more appliances for such traffic controlling application, than only libvirt, standard API will make it possible to reuse it in other solutions — be it python implementation, C or even erlang.
Python was choosen 'cause of one simple reason: it allows to implement netlink message encoder/decoder a way much easier than C, and since the netlink control message communication doesn't require high throughput, python's performance more than enough. Simple structure handling in python allows in this case really fast time-to-market features delivery, sometimes it makes sense. Beside of that, some high-level virtualization solutions uses python anyway, so mostly this application will not even create unique package dependencies, using only python stdlib and no tricks like ctypes and so on.
But as I said, the language in this case really doesn't mean so much, having standard RPC API. C — ok, maybe someone will implement it in C also. The key issue is to have the API standard.
Yes, the choice language *today* does matter, because it impacts on the minimum install base size possible with libvirt. It also impacts on operational aspects becasue python is not a low footprint runtime from a memory POV, and not OOM friendly. As per my previous mail I also think this kind of higher level interface to netlink should be a C library rather than an RPC system. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 05/20/2013 11:21 AM, Daniel P. Berrange wrote:
Yes, the choice language *today* does matter, because it impacts on the minimum install base size possible with libvirt. It also impacts on operational aspects becasue python is not a low footprint runtime from a memory POV, and not OOM friendly. As per my previous mail I also think this kind of higher level interface to netlink should be a C library rather than an RPC system.
I also would like to create a C lib with some optional high-level bindings. But as I think, it is simply not possible in nearest time, it is too hard, I know the amount and state of code of the `tc` utility. Right now iproute's `tc` is the only complete implementation of the traffic control messages (and it is a total mess-up with tons of non-actual legacy), the second shortly will be the one I write (in pure python, yes). I would like to see another one, but I just do not expect it in some foreseeable future. Ok, time will show — maybe I'm totally wrong. Another approach could be providing basic functionality just as it is done right now — calling the external utility (from my point of view, it is not better than RPC) and to make pluggable network management politics, that could be provided by loadable C libraries. It would give some space for extensibility. -- Peter V. Saveliev
participants (3)
-
Daniel P. Berrange -
Michal Privoznik -
Peter V. Saveliev