Re: [libvirt] Applying for GSoC project 'Introducing job control to the storage driver' in libvirt

On 09.03.2015 07:29, Taowei Luo wrote: > Hello, everyone. > Hey! > I'm Taowei Luo. A participator in Google Summer of Code last year for > the project rewriting vbox driver in libvirt. I am so glad that my > project isn't found in this year's GSoC project list :) > > Having a nice experience last year, libvirt is my top option to join > the GSoC 2015. And my interesting project is “Introducing job control > to the storage driver”. It's nice to see people returning. > > I have read some materials about this project. Including I noticed > that Tucker had attempted it last year. It seems he didn't make it > through the final evaluation. During his working, some discussion were > made in mail-list. It is really inspiring to me. So I have some basic > ideas for now. > > The project requires job control functions (at least, reporting the > progress) in storage driver. Obviously, it contains two part. First > find codes that really do the storage work which may take a long > period and can be asynchronized. Then, extract it to the job control > part so make it under the control. It would not be hard to find codes > that really need asynchronization. Maybe by dumping the debug message > and tracing the function calls. So the big part is design the APIs for > job control. I think the goal for API design is not only make it > workable but also make it reasonable and extendable. The idea is to turn qemuDomainObj*Job* a separate module, that would work over virObject. Subsequently, we can add BeginJob and EndJob APIs into storage driver too. Moreover, places where libvirt monitors progress of a storage operation, we can fill in jobInfo strucut (although, that one in qemu is not generic enough I guess, so maybe we need to invent a new one). Then, CancelJob API can be implemented in storage driver too. > > To achieve this, a lot of details were discussed in last year's > proceeding. I summered it and add my own opinion. > > Parallel jobs: The idea result is that several jobs work in parallel > and libvirt monitors and controls it. There are two ways for parallel: > thread and process. I prefer process. In process, we can easier > implement the idea of *control* by signal. Process has better > independence than thread. What's more, it is a low coupling design. Libvirt already uses that. virCommandExec* is to be find all over the storage driver. > > Synchronization: process can use system level lock to make sure it > obtain the resources. If the process can't obtain it could exit with > failure (or wait). In process, we can leave most resource competition > handling by OS. If thread is used instead, we need to think about > resource competition between libvirt and other process, and at the > same time, those competitions in libvirt thread. > > Management: We execute those asynchronous codes in a new process. In > libvirt, it invokes those processes with parsed arguments. Libvirtd > would have a process pool to store the pids and some attached > information for each process. Signal would be used to communicate > between process and libvirt. > > Expandability: Some other jobs like domain migration could be in > implemented under this design. It's all about creating new jobs with > parsed arguments, which tells the child process what to do and what > resources they need. > > Privacy: If new jobs are created in process, user may access the > process directly and not noticing libvirt. Function sigaction(), which > provides the pid of sending process, could be used to register > responding functions. Meanwhile, atexit() register functions that > execute when the process is going to exit. It is helpful on notifying > libvirtd the end of the job. This is already handled by virCommand subsystem. > > What already have: Besides the discussion, some other resources would > be helpful for this project. qemu has a prototype for job control in > migration. We already have gnulib and tools in util/virprocess.h and > util/virthread.h to achieve parallel. > > When undertaking, I would firstly implement the job control part. It > would have some basic functions. Make one storage API work in it. > Then, adding job control support for the rest APIs. > > This is all about what I came up with. Maybe those ideas are old and > repeating. But I think it is a workable plan. Waiting for feedback. Majority looks reasonable to me. Michal