On Fri, Oct 03, 2008 at 06:22:27PM +0200, Daniel Veillard wrote: I did respond back to that email at http://www.mail-archive.com/libvir-list@redhat.com/msg08541.html Since then patches have been merged which clean up that part of the code. Rawhide has a newer package, and I am working on packaging up v0.32 for rawhide now. Should be pushed out sometime soon. If you would not mind, could you take a look at the latest snapshot available at http://sourceforge.net/projects/libcg , and let us know what is missing, we can implement it so that libvirt's needs are met. Thanks, -- regards, Dhaval

On Fri, Oct 03, 2008 at 09:31:52PM +0530, Balbir Singh wrote: So from a functional point of view you are addressing essentially three use cases 1. System configuration for controllers 2. Automatic task classification 3. Application development API for creating groups If each piece is correctly designed, the choice of implementation for each of these can be, and in some cases must be, totally independant. Since the kernel restricts that a single controller can only be attached to one cgroupsfs mount point, and one attach cannot be changed, the choice of how / where to mount controllers must remain outside the scope of applications. If any application using cgroups were to specify mount points, it would be inflicting its own requirements on every user of cgroups. This implies that applications must be designed to work with whatever controller mount configuration the admin has configured, and not configure stuff themselves. So impl for point 1 (configuration) must, by neccessity, be completely independant of impl for point 3 (application API). Considering automatic task classification. The task classification engine must be able to cope with the fact that applications have some functional requirements on cgroups setup. Taking libvirt as an example, we have a specific need to apply some controllers over a group of processes forming a container. A task classification engine must not re-clasify individual tasks within a container because that would conflict with the semantics required by libvirt. It is, however, free to re-classify the libvirtd daemon itself - whatever cgroup libvirtd is placed in, it will create the LXC cgroups below this point. So if libvirt is designed correctly, it will work with whatever cgroup task classification engine that might be running. Similarly if the task classification engine has been designed to co-operate with applications there is no problem running it alonside libvirt. Thus the implementation of points 2 (task classification) and point 3 (application API) have no need to be formally tied together. Furthermore tieing them together does not magically solve the problem that both applications & the cgroups task classification engine need to be intelligently designed to co-operate. This is important if the library code is providing significant value add to the application using it. As it stands, libcgroup is merely a direct interface to the cgroups filesystem providing weakly typed setters & getters - with the exception of looking at the mount table to find where a controller lives, this is not hard / complex code, so the benefits of re-use are not particularly high. In such a scenario reducing code duplication is not in itself a benefit, since there are costs associated with using external libraries. It is more complicated integrate 2 independant style sof API, particularly with different views on error reporting, memory management and varying expectations for the semantic models exposed. There are a number of 'hard' questions wrt to cgroups usage by applications, two of which are outlined above. Simply having all applications use a single API cannot magically solve any of these problems - no matter what API is used application developers need to take care to design their usage of cgroups such that it 'plays nicely' with other applications. The use of cgroups is an internal implementation detail for libvirt's LXC driver. In comon with all libvirt drivers, the user has no need to know about the underlying impl details and these can & will change at will as we discover better ways to achieve things. As such its irrelevant to a user how we configure the cgroups filesytem for libvirt - whether we do it directly, or via libcg the end result is identical - a set of directories in the cgroups filesystem. Our of the 3 functional areas I outlined earlier on, the ones that can provide the most value to users of libvirt, are a standardized means to configure cgroups mounts and have them active at boot time, and the task classification engine. Both of these are interesting & hard problems for which there is clear value in only having 1 global implementation. Neither of these has a strong dependancy on the API that 3rd party applications use to talk to cgroups filesytem - from a functional point of view they ought to 'just work' no matter what API the app uses, provided the application is design to assume presence of other applications using cgroups. So I do see value in the work cgroups project is producing, but the application development API is the least critical part of this - the task engine and cofiguration policy is the key value add that is useful to everyone. Since cgroups is an internal implementation detail for libvirt LXC driver, we can change our impl at any time we like, should circumstance change & use of libcgroup.so be critically neccessary. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Fri, Oct 03, 2008 at 09:31:52PM +0530, Balbir Singh wrote: To expand on what I said in my other mail about providing value-add over the representation exposed by the kernel, here's some thoughts on the API exposed. Consider the following high level use case of libvirt - A set of groups, in a 3 level hierarchy <APPNAME>/<DRIVER>/<DOMAIN> - Control the ACL for block/char devices - Control memory limits This translates into an underling implementation, that I need to create 3 levels of cgroups in the filesystem, attach my PIDs at the 3rd level use the memory and device controllers and attach PIDs at the 3rd, and set values for attributes exposed by the controllers. Notice I'm not actually setting any config parms at the 1st & 2nd levels, but they do need to still exist to ensure namespace uniqueness amongst different applications using cgroups. The current cgroups API provides APIs that directly map to individual actions wrt the kernel filesystem exposed. So as an application developer I have to explicitly create the 3 levels of hierarchy, tell it I want to use memory & device controllers, format config values into the syntax required for each attribute, and remeber the attribute names. // Create the hierachy <APPNAME>/<DRIVER>/<DOMAIN> c1 = cgroup_new_cgroup("libvirt") c2 = cgroup_new_cgroup_parent(c1, "lxc") c3 = cgroup_new_cgroup_parent(c2, domain.name) // Setup the controllers I want to use cgroup_add_controler(c3, "devices") cgroup_add_controller(c3, "memory") // Add my domain's PID to the cgroup cgroup_attach_task(c3, domain.pid) // Set the device ACL limits cgroup_set_value_string(c2, "devices.deny", "a"); char buf[1024]; sprintf(buf, "%c %d:%d", 'c', 1, 3); cgroup_set_value_stirng(c2, "devices.allow", buf); // Set memory limit cgroup_set_value_uint64(c2, "memory.limit_in_bytes", domain.memory * 1024); This really isn't providing any semantically useful abstraction over the direct filesytem manipulation. Just a bunch of wrappers for mkdir(), mount() and read()/write() calls. My application still has to know far too much information about the details of cgroups as exposed by the kernel. I do not care that there is a concept of 'controllers' at all, I just want to set device ACLs and memory limits. I do not care what the attributes in the filesystem are called, again I just want to set device ACLs and memory limits. I do not care what the data format for them must be for device/memory settings. Memory settings could be stored in base-2, base-10 or base-16 I should not have to know this information. With this style of API, the library provide no real value-add or compelling reason to use it. What might a more useful API look like? At least from my point of view, I'd like to be able to say: // Tell it I want $PID placed in <APPNAME>/<DRIVER>/<DOMAIN> char *path[] = { "libvirt", "lxc", domain.name}; cg = cgroup_new_path(path, domain.pid) // I want to deny all devices cgroup_deny_all_devices(cg); // Allow /dev/null - either by node/major/minor cgroup_allow_device_node(cg, 'c', 1, 3); // Or more conviently just give it a node to copy info from cgroup_allow_device_node(cg, "/dev/null") // Set memory in KB cgroup_set_memory_limit_kb(cg, domain.memory) Notice how with such a style of API, I don't need to know anything about the low level implementation details - I'm working entirely in terms of semantically meaningful concepts. Now, comes the hard bit - you have to figure out what semantic concepts you want to expose to applications. The example here would be suitable for libvirt, but not neccessarily for other applications. Picking the right APIs is very much much harder than just exposing the kernel capabilities directly as libcgroup.h does now, but the trade off is that the resulting API would be much more useful and interesting to app developers. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Fri, Oct 3, 2008 at 11:43 PM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote: True, it definitely does and the way I look at APIs is that they are layers. We've built the first layer that abstracts permissions, paths and strings into a set of useful API. The second layer does things that you say, the question then is why don't we have it yet? Let me try and answer that question 1. We've been trying to build configuration, classification and the low level plumbing 2. We've been planning to build the exact same thing that you say, we call that the pluggable architecture, where controller plug in their logic and provide the abstractions you need, but not gotten there yet. When you announced cgroup support in libvirt, it was definitely going to be a user and we hoped that you would come to us with your exact requirements that you've mentioned now (believe me, your feedback is very useful). The question then to ask is, is it cheaper for you to build these abstractions into libvirt or either helped us or asked us to do so, we would have gladly obliged. You might say that the onus is on the maintainers to do the right thing without feedback, but I would beg to differ. What you've asked for, I consider as a layer on top of the API we have now and should be easy to build. Yes, I agree this is definitely more usable and friendlier. These are not hard to do implement today, in fact implementing them would require a few calls to existing API and can be built as controller specific layers (I call them as plugins for each controller). I like what you've proposed very much and I am going to start building these abstractions and make them available in libcgroup. At some point, I hope you will find them useful enough so as to drop your abstractions (which I would hope you had directly built into libcgroup and used, so that more people would have benefited from it) and use them. Balbir

On Sat, Oct 4, 2008 at 1:17 AM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote: Please remember my words "if you ever find that you have a code base that looks like what we have in libcgroups, please remember to switch over to libcgroup". I fear that you will reach that stage, the code that is going in right now has too many things hard-coded and will need a lot of changes going forward, things like adding support for new controllers is not going to be straight forward, your assumption that only root can create a container might be broken and we'll build support for hierarchies, which will require further changes, etc. I am not scaring you, just trying to make sure we don't solve the same problems twice. Balbir