Re: [libvirt] Using virtEvents

On 12/29/2011 04:03 AM, D. Herrendoerfer wrote: [Caveat - I didn't write the threading model used in libvirt, so my answers are my interpretation of the code as I have come to understand it, but might not be 100% accurate] I'm not quite sure what you were asking, or which processes vs. threads you are viewing. Remember, in Linux, threads and processes share the same namespace of thread ids (that is, a process id in Linux is the thread-id of its primary thread), and the difference between a single-threaded app and a multi-threaded app is whether the process owns more than one thread id. Calling fork() creates a new process (and thus a new thread), and the process of starting libvirtd ends up calling fork() twice to get the daemonization right, so that may explain the "2 separate threads running that are forked". Once the daemon is up and running, it becomes multi-threaded, by spawning helper threads all within the same process, which would explain the "4 more that appear to be threads". Basically, libvirtd is set up in an event loop, where the main thread exists solely to monitor shutdown requests, and all other threads do round-robin servicing of RPC events that come in over the sockets. Threads aren't forked, only processes are (but forking a new process creates a new thread, mainly the primary thread of the new process). Also, processes do not share memory, but threads within a process do, so a polling loop in a main thread _does_ have access to worker threads in the same process. It appears that you have some confusion in the terminology, which is in turn making it hard for me to know if I'm answering your questions. This question added in virsh, which adds another wrinkle into the picture - virsh is often capable of doing its job as a single-threaded application, which connects to the libvirtd socket to request libvirtd to do something with the qemu hypervisor. However, virsh is also designed for multi-threaded tasks - in particular, the ability to start a long-running task such as a migration, and cancel it midway, requires that virsh have a couple of threads, the primary thread that processes user commands delegates to a helper thread that interacts with libvirtd as well as listening for Ctrl-C events from the user or for migration completion events from libvirtd. This means that both libvirtd _and_ virsh are using an event loop - libvirtd to listen for incoming activity on the sockets as well as completion of those activities by the hypervisor, and virsh to listen for incoming activity from libvirtd completion. So in true code reuse fashion, we reuse the event loop in both processes, and you thus have to be careful which process you are asking about when discussing threads that are handling various events. The event loop driver supports both file-based events, such as recognizing when traffic arrives on the public libvirtd socket which needs handling by one of the RPC worker threads, as well as time-based events. But it is not limited to the public libvirtd socket; the qemu hypervisor driver can register additional fd's to be watched, such as the monitor socket tied to the qemu running each domain. As long as you can come up with a scenario where an fd can track a change, then you can register that fd with the existing event loop. And don't forget that you can use pipes as a way to set up fds (after all, our SIGINT handler for Ctrl-C in virsh is nothing more than a pipe-to-self so that we have an fd to wake up the event loop). So it sounds like you are asking how to make a client, such as virsh, aware that an event was recognized in libvirtd. At this point, it boils down to an RPC event - whenever something in libvirtd fires that a client might usefully want to learn about, we can then map that to the RPC event mechanisms. Look at the virConnectDomainEventRegisterAny API, which is a means for setting up a callback function in the client that will be called when the event happens in libvirtd, by means of an RPC message from libvirtd to the client containing the details appropriate for the callback. Also, docs/internals/rpc.html describes the details behind RPC event calls; normally, things are driven by the client (client issues a call, server issues a response), but in the case of events, it is an asynchronous call where the server issues the event, and the client reacts to the event by calling any registered callback functions. I'm not sure if I'm on the right track for answering what you are really trying to do, but hopefully it helped, or someone else can chime in to correct anything I got wrong. -- Eric Blake eblake(a)redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org