On Thu, Dec 29, 2011 at 12:03:06PM +0100, D. Herrendoerfer wrote: There are quite a few threads in libvirtd, so lets go from the beginning. When libvirtd starts up, it is obviously single threaded. One of the first things we do is to create the server state object - virNetServerPtr. This object is what processes for incoming client connections, creating virNetServerClientPtr instances. The virNetServerPtr is also responsible for dispatching RPC messages. To allow maximum concurrency, the virNetServerPtr creates two pools of worker threads for handling RPC. One pool is the so called "high priority" threads pool, which handles RPC messages which are very fast and never block. The other thread pool handles all other RPC messages, including those which may take a long time. Each of these pools has an initial 5 threads, so we have 10 threads in total for RPC workers, but this number may grow over time - the thread limits are controlled in libvirtd.conf During initialization, we need to run the global driver init functions (eg qemudStartup and friends). This is done by spawning a new thread, which invokes virStateInit(), waits for it to complete and then exits. The initial process leader thread, invokes virNetServerRun() which runs the main event loop. So most of the time there the 10+ worker threads, plus the event loop thread. At times other threads may be invoked, eg during tunnelled migration the QEMU driver spawns an extra thread to handle I/O. The NWfilter code spawns other threads to handle certain operations. Also core dumps on crashing QEMU will spawn threads. virsh is a completely separate process from libvirtd. Any event loop callback must be sure to complete in very fast time, since all I/O processsing is blocked while an event callback is running. So typically callbacks would restrict themselvs to reading/writing I/O on the triggering file descriptor, or some other small non-blocking piece of worker. If your event loop callback intends to invoke complex APis, which may in turn need to do I/O, then you must spawn a new thread so that you don't block the whole event loop. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On 12/29/2011 04:03 AM, D. Herrendoerfer wrote: [Caveat - I didn't write the threading model used in libvirt, so my answers are my interpretation of the code as I have come to understand it, but might not be 100% accurate] I'm not quite sure what you were asking, or which processes vs. threads you are viewing. Remember, in Linux, threads and processes share the same namespace of thread ids (that is, a process id in Linux is the thread-id of its primary thread), and the difference between a single-threaded app and a multi-threaded app is whether the process owns more than one thread id. Calling fork() creates a new process (and thus a new thread), and the process of starting libvirtd ends up calling fork() twice to get the daemonization right, so that may explain the "2 separate threads running that are forked". Once the daemon is up and running, it becomes multi-threaded, by spawning helper threads all within the same process, which would explain the "4 more that appear to be threads". Basically, libvirtd is set up in an event loop, where the main thread exists solely to monitor shutdown requests, and all other threads do round-robin servicing of RPC events that come in over the sockets. Threads aren't forked, only processes are (but forking a new process creates a new thread, mainly the primary thread of the new process). Also, processes do not share memory, but threads within a process do, so a polling loop in a main thread _does_ have access to worker threads in the same process. It appears that you have some confusion in the terminology, which is in turn making it hard for me to know if I'm answering your questions. This question added in virsh, which adds another wrinkle into the picture - virsh is often capable of doing its job as a single-threaded application, which connects to the libvirtd socket to request libvirtd to do something with the qemu hypervisor. However, virsh is also designed for multi-threaded tasks - in particular, the ability to start a long-running task such as a migration, and cancel it midway, requires that virsh have a couple of threads, the primary thread that processes user commands delegates to a helper thread that interacts with libvirtd as well as listening for Ctrl-C events from the user or for migration completion events from libvirtd. This means that both libvirtd _and_ virsh are using an event loop - libvirtd to listen for incoming activity on the sockets as well as completion of those activities by the hypervisor, and virsh to listen for incoming activity from libvirtd completion. So in true code reuse fashion, we reuse the event loop in both processes, and you thus have to be careful which process you are asking about when discussing threads that are handling various events. The event loop driver supports both file-based events, such as recognizing when traffic arrives on the public libvirtd socket which needs handling by one of the RPC worker threads, as well as time-based events. But it is not limited to the public libvirtd socket; the qemu hypervisor driver can register additional fd's to be watched, such as the monitor socket tied to the qemu running each domain. As long as you can come up with a scenario where an fd can track a change, then you can register that fd with the existing event loop. And don't forget that you can use pipes as a way to set up fds (after all, our SIGINT handler for Ctrl-C in virsh is nothing more than a pipe-to-self so that we have an fd to wake up the event loop). So it sounds like you are asking how to make a client, such as virsh, aware that an event was recognized in libvirtd. At this point, it boils down to an RPC event - whenever something in libvirtd fires that a client might usefully want to learn about, we can then map that to the RPC event mechanisms. Look at the virConnectDomainEventRegisterAny API, which is a means for setting up a callback function in the client that will be called when the event happens in libvirtd, by means of an RPC message from libvirtd to the client containing the details appropriate for the callback. Also, docs/internals/rpc.html describes the details behind RPC event calls; normally, things are driven by the client (client issues a call, server issues a response), but in the case of events, it is an asynchronous call where the server issues the event, and the client reacts to the event by calling any registered callback functions. I'm not sure if I'm on the right track for answering what you are really trying to do, but hopefully it helped, or someone else can chime in to correct anything I got wrong. -- Eric Blake eblake(a)redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org