[GSoC] Interested Student for the Project: 'Introducing job control to the storage driver'
Hi Everyone, I'm Prathamesh Chavan, a final year student at studying Computer Science and Engineering at IIT Kharagpur. I've been part of GSoC'17 under the Git Organization and have an internship experience at Nutanix, India during last summer. I'm also currently working on a tired file system with software wear management for nvm technologies as my master project. I was interested in contributing to the project: "Introducing job control to the storage driver" under Google Summer of Code - 2020. I was currently going through the codebase and also was successfully able to build it. It would be great if I was assigned a byte-task regarding the same which could help me understand the project better, Thanks, Prathamesh Chavan
On 23. 3. 2020 10:45, Prathamesh Chavan wrote:
Hi Everyone,
Hey! It's nice to see somebody new interested in libvirt.
I'm Prathamesh Chavan, a final year student at studying Computer Science and Engineering at IIT Kharagpur. I've been part of GSoC'17 under the Git Organization and have an internship experience at Nutanix, India during last summer. I'm also currently working on a tired file system with software wear management for nvm technologies as my master project. I was interested in contributing to the project: "Introducing job control to the storage driver" under Google Summer of Code - 2020. I was currently going through the codebase and also was successfully able to build it. It would be great if I was assigned a byte-task regarding the same which could help me understand the project better,
There is no assignment process, just pick anything you want. Adapting to GLib is pretty popular as it usually removes some lines :-) But feel free to pick anything you would like. And for the GSoC project itself; we currently have in_use member for virStorageVolDef structure which serves as a refcounter. Then we have some mutexes to guard access to the counter. Whenever a 'long running' storage API is called (e.g. virStorageVolWipe(), virStorageVolDownload(), ...) the refcounter is increased at the beginning and then decreased at the end. Then, virStorageVolDelete() checks this refcounter to see if there is some other thread using it. But we already have something similar implemented for virDomainObj - jobs. Whenever an API wants to work with a domain object [1], it shall acquire a job. This job then prevents other threads from modifying the object meanwhile The threads wanting to work over the same object will serialize. The whole job control is described in src/qemu/THREADS.txt so I would start there. Michal 1: actually, that is not entirely true. Acquiring a job is required only for those APIs which want to unlock the domain object in the middle. Therefore, some 'short' APIs have no job acquiring implemented (e.g. qemuDomainGetInfo()). But some other 'short' APIs do (e.g. qemuDomainReset()).
On Wed, Mar 25, 2020 at 02:59:16PM +0100, Michal Prívozník wrote:
On 23. 3. 2020 10:45, Prathamesh Chavan wrote:
Hi Everyone,
Hey!
It's nice to see somebody new interested in libvirt.
I'm Prathamesh Chavan, a final year student at studying Computer Science and Engineering at IIT Kharagpur. I've been part of GSoC'17 under the Git Organization and have an internship experience at Nutanix, India during last summer. I'm also currently working on a tired file system with software wear management for nvm technologies as my master project. I was interested in contributing to the project: "Introducing job control to the storage driver" under Google Summer of Code - 2020. I was currently going through the codebase and also was successfully able to build it. It would be great if I was assigned a byte-task regarding the same which could help me understand the project better,
There is no assignment process, just pick anything you want. Adapting to GLib is pretty popular as it usually removes some lines :-) But feel free to pick anything you would like.
And for the GSoC project itself; we currently have in_use member for virStorageVolDef structure which serves as a refcounter. Then we have some mutexes to guard access to the counter. Whenever a 'long running' storage API is called (e.g. virStorageVolWipe(), virStorageVolDownload(), ...) the refcounter is increased at the beginning and then decreased at the end. Then, virStorageVolDelete() checks this refcounter to see if there is some other thread using it.
But we already have something similar implemented for virDomainObj - jobs. Whenever an API wants to work with a domain object [1], it shall acquire a job. This job then prevents other threads from modifying the object meanwhile The threads wanting to work over the same object will serialize. The whole job control is described in src/qemu/THREADS.txt so I would start there.
Michal
1: actually, that is not entirely true. Acquiring a job is required only for those APIs which want to unlock the domain object in the middle. Therefore, some 'short' APIs have no job acquiring implemented (e.g. qemuDomainGetInfo()). But some other 'short' APIs do (e.g. qemuDomainReset()).
What if you have a function which changes something, or even looks up something and it does not need to communicate with QEMU at all, but it is called when some other async job is waiting for a monitor. I know we discussed this multiple times and I always forget some minor detail, so forgive me if I'm asking this for 27th time already. But shouldn't that function also have a job? Another question is if this should also be the case for just reading and returning some info about the domain, shouldn't it also take a job, even if it is just a QEMU_JOB_QUERY? Otherwise we'd have to make sure that before each unlock of a domain (even during a job) all the data integrity is kept. Martin
On 29. 3. 2020 0:43, Martin Kletzander wrote:
On Wed, Mar 25, 2020 at 02:59:16PM +0100, Michal Prívozník wrote:
On 23. 3. 2020 10:45, Prathamesh Chavan wrote:
Hi Everyone,
Hey!
It's nice to see somebody new interested in libvirt.
I'm Prathamesh Chavan, a final year student at studying Computer Science and Engineering at IIT Kharagpur. I've been part of GSoC'17 under the Git Organization and have an internship experience at Nutanix, India during last summer. I'm also currently working on a tired file system with software wear management for nvm technologies as my master project. I was interested in contributing to the project: "Introducing job control to the storage driver" under Google Summer of Code - 2020. I was currently going through the codebase and also was successfully able to build it. It would be great if I was assigned a byte-task regarding the same which could help me understand the project better,
There is no assignment process, just pick anything you want. Adapting to GLib is pretty popular as it usually removes some lines :-) But feel free to pick anything you would like.
And for the GSoC project itself; we currently have in_use member for virStorageVolDef structure which serves as a refcounter. Then we have some mutexes to guard access to the counter. Whenever a 'long running' storage API is called (e.g. virStorageVolWipe(), virStorageVolDownload(), ...) the refcounter is increased at the beginning and then decreased at the end. Then, virStorageVolDelete() checks this refcounter to see if there is some other thread using it.
But we already have something similar implemented for virDomainObj - jobs. Whenever an API wants to work with a domain object [1], it shall acquire a job. This job then prevents other threads from modifying the object meanwhile The threads wanting to work over the same object will serialize. The whole job control is described in src/qemu/THREADS.txt so I would start there.
Michal
1: actually, that is not entirely true. Acquiring a job is required only for those APIs which want to unlock the domain object in the middle. Therefore, some 'short' APIs have no job acquiring implemented (e.g. qemuDomainGetInfo()). But some other 'short' APIs do (e.g. qemuDomainReset()).
What if you have a function which changes something, or even looks up something and it does not need to communicate with QEMU at all, but it is called when some other async job is waiting for a monitor. I know we discussed this multiple times and I always forget some minor detail, so forgive me if I'm asking this for 27th time already. But shouldn't that function also have a job?
Oh yeah, I wasn't trying to be exhaustive and cover all corner cases when introducing jobs to somebody who hears about the concept for the first time. If a function modifies internal state then it should grab _MODIFY job. This is, as you correctly say, to mutually exclude with another thread that is doing some other modification. But frankly, I don't remember all the details. I recall discussing whether each API should grab a job, but I just don't remember the details.
Another question is if this should also be the case for just reading and returning some info about the domain, shouldn't it also take a job, even if it is just a QEMU_JOB_QUERY? Otherwise we'd have to make sure that before each unlock of a domain (even during a job) all the data integrity is kept.
I'm not completely sure what you are asking here. And how it relates to storage driver. Michal
On Mon, Mar 30, 2020 at 12:23:32PM +0200, Michal Prívozník wrote:
On 29. 3. 2020 0:43, Martin Kletzander wrote:
On Wed, Mar 25, 2020 at 02:59:16PM +0100, Michal Prívozník wrote:
On 23. 3. 2020 10:45, Prathamesh Chavan wrote:
Hi Everyone,
Hey!
It's nice to see somebody new interested in libvirt.
I'm Prathamesh Chavan, a final year student at studying Computer Science and Engineering at IIT Kharagpur. I've been part of GSoC'17 under the Git Organization and have an internship experience at Nutanix, India during last summer. I'm also currently working on a tired file system with software wear management for nvm technologies as my master project. I was interested in contributing to the project: "Introducing job control to the storage driver" under Google Summer of Code - 2020. I was currently going through the codebase and also was successfully able to build it. It would be great if I was assigned a byte-task regarding the same which could help me understand the project better,
There is no assignment process, just pick anything you want. Adapting to GLib is pretty popular as it usually removes some lines :-) But feel free to pick anything you would like.
And for the GSoC project itself; we currently have in_use member for virStorageVolDef structure which serves as a refcounter. Then we have some mutexes to guard access to the counter. Whenever a 'long running' storage API is called (e.g. virStorageVolWipe(), virStorageVolDownload(), ...) the refcounter is increased at the beginning and then decreased at the end. Then, virStorageVolDelete() checks this refcounter to see if there is some other thread using it.
But we already have something similar implemented for virDomainObj - jobs. Whenever an API wants to work with a domain object [1], it shall acquire a job. This job then prevents other threads from modifying the object meanwhile The threads wanting to work over the same object will serialize. The whole job control is described in src/qemu/THREADS.txt so I would start there.
Michal
1: actually, that is not entirely true. Acquiring a job is required only for those APIs which want to unlock the domain object in the middle. Therefore, some 'short' APIs have no job acquiring implemented (e.g. qemuDomainGetInfo()). But some other 'short' APIs do (e.g. qemuDomainReset()).
What if you have a function which changes something, or even looks up something and it does not need to communicate with QEMU at all, but it is called when some other async job is waiting for a monitor. I know we discussed this multiple times and I always forget some minor detail, so forgive me if I'm asking this for 27th time already. But shouldn't that function also have a job?
Oh yeah, I wasn't trying to be exhaustive and cover all corner cases when introducing jobs to somebody who hears about the concept for the first time. If a function modifies internal state then it should grab _MODIFY job. This is, as you correctly say, to mutually exclude with another thread that is doing some other modification.
But frankly, I don't remember all the details. I recall discussing whether each API should grab a job, but I just don't remember the details.
Another question is if this should also be the case for just reading and returning some info about the domain, shouldn't it also take a job, even if it is just a QEMU_JOB_QUERY? Otherwise we'd have to make sure that before each unlock of a domain (even during a job) all the data integrity is kept.
I'm not completely sure what you are asking here. And how it relates to storage driver.
I'm lust trying to minimize the complexity and say that it is the best if all APIs working with an object should take a job, no exceptions (to make it simpler to follow). I think we're on the same boat ;-)
Michal
On Sun, Mar 29, 2020 at 5:13 AM Martin Kletzander <mkletzan@redhat.com> wrote:
On Wed, Mar 25, 2020 at 02:59:16PM +0100, Michal Prívozník wrote:
On 23. 3. 2020 10:45, Prathamesh Chavan wrote:
Hi Everyone,
Hey!
It's nice to see somebody new interested in libvirt.
I'm Prathamesh Chavan, a final year student at studying Computer Science and Engineering at IIT Kharagpur. I've been part of GSoC'17 under the Git Organization and have an internship experience at Nutanix, India during last summer. I'm also currently working on a tired file system with software wear management for nvm technologies as my master project. I was interested in contributing to the project: "Introducing job control to the storage driver" under Google Summer of Code - 2020. I was currently going through the codebase and also was successfully able to build it. It would be great if I was assigned a byte-task regarding the same which could help me understand the project better,
There is no assignment process, just pick anything you want. Adapting to GLib is pretty popular as it usually removes some lines :-) But feel free to pick anything you would like.
And for the GSoC project itself; we currently have in_use member for virStorageVolDef structure which serves as a refcounter. Then we have some mutexes to guard access to the counter. Whenever a 'long running' storage API is called (e.g. virStorageVolWipe(), virStorageVolDownload(), ...) the refcounter is increased at the beginning and then decreased at the end. Then, virStorageVolDelete() checks this refcounter to see if there is some other thread using it.
But we already have something similar implemented for virDomainObj - jobs. Whenever an API wants to work with a domain object [1], it shall acquire a job. This job then prevents other threads from modifying the object meanwhile The threads wanting to work over the same object will serialize. The whole job control is described in src/qemu/THREADS.txt so I would start there.
Michal
1: actually, that is not entirely true. Acquiring a job is required only for those APIs which want to unlock the domain object in the middle. Therefore, some 'short' APIs have no job acquiring implemented (e.g. qemuDomainGetInfo()). But some other 'short' APIs do (e.g. qemuDomainReset()).
What if you have a function which changes something, or even looks up something and it does not need to communicate with QEMU at all, but it is called when some other async job is waiting for a monitor. I know we discussed this multiple times and I always forget some minor detail, so forgive me if I'm asking this for 27th time already. But shouldn't that function also have a job? Another
I think the API currently allows normal jobs in the presence of another async ongoing job (given the normal job is compatible with the async one). And I think it was this commit (qemu: Allow all query commands to be run during long jobs) [1], which allowed us to query (or run synchronous jobs) while there is an async long-running job going on. Also, regarding the GSoC Project, you mentioned earlier that currently, a reference counter is being used in the storage driver. What the project will aim to rather remove this reference counter and implement a job control routine as done wrt the domain jobs in this commit (qemu: completely rework reference counting) [2]. I've gone through the way the domain jobs are being handled in qemu. And right now I'm going through the storage drivers of libvirt, particularly the way virStorageVolDef is being handled, and how its values are read and changed. If there is anything else, I should be looking into, any suggestions of this sort would be really helpful. Thanks, Prathamesh Chavan [1]: https://github.com/libvirt/libvirt/commit/361842881e0 [2]: https://github.com/libvirt/libvirt/commit/540c339a253
participants (3)
-
Martin Kletzander -
Michal Prívozník -
Prathamesh Chavan