[libvirt] Applying for GSoC project 'Introducing job control to the storage driver' in libvirt

Hello, everyone. I'm Taowei Luo. A participator in Google Summer of Code last year for the project rewriting vbox driver in libvirt. I am so glad that my project isn't found in this year's GSoC project list :) Having a nice experience last year, libvirt is my top option to join the GSoC 2015. And my interesting project is “Introducing job control to the storage driver”. I have read some materials about this project. Including I noticed that Tucker had attempted it last year. It seems he didn't make it through the final evaluation. During his working, some discussion were made in mail-list. It is really inspiring to me. So I have some basic ideas for now. The project requires job control functions (at least, reporting the progress) in storage driver. Obviously, it contains two part. First find codes that really do the storage work which may take a long period and can be asynchronized. Then, extract it to the job control part so make it under the control. It would not be hard to find codes that really need asynchronization. Maybe by dumping the debug message and tracing the function calls. So the big part is design the APIs for job control. I think the goal for API design is not only make it workable but also make it reasonable and extendable. To achieve this, a lot of details were discussed in last year's proceeding. I summered it and add my own opinion. Parallel jobs: The idea result is that several jobs work in parallel and libvirt monitors and controls it. There are two ways for parallel: thread and process. I prefer process. In process, we can easier implement the idea of *control* by signal. Process has better independence than thread. What's more, it is a low coupling design. Synchronization: process can use system level lock to make sure it obtain the resources. If the process can't obtain it could exit with failure (or wait). In process, we can leave most resource competition handling by OS. If thread is used instead, we need to think about resource competition between libvirt and other process, and at the same time, those competitions in libvirt thread. Management: We execute those asynchronous codes in a new process. In libvirt, it invokes those processes with parsed arguments. Libvirtd would have a process pool to store the pids and some attached information for each process. Signal would be used to communicate between process and libvirt. Expandability: Some other jobs like domain migration could be in implemented under this design. It's all about creating new jobs with parsed arguments, which tells the child process what to do and what resources they need. Privacy: If new jobs are created in process, user may access the process directly and not noticing libvirt. Function sigaction(), which provides the pid of sending process, could be used to register responding functions. Meanwhile, atexit() register functions that execute when the process is going to exit. It is helpful on notifying libvirtd the end of the job. What already have: Besides the discussion, some other resources would be helpful for this project. qemu has a prototype for job control in migration. We already have gnulib and tools in util/virprocess.h and util/virthread.h to achieve parallel. When undertaking, I would firstly implement the job control part. It would have some basic functions. Make one storage API work in it. Then, adding job control support for the rest APIs. This is all about what I came up with. Maybe those ideas are old and repeating. But I think it is a workable plan. Waiting for feedback. Regards, Taowei