On Thu, Dec 29, 2011 at 12:03:06PM +0100, D. Herrendoerfer wrote:

Sorry to come back on this.

How are the different threads of the libvirt daemon separated ? I understand that there are 2 separate threads running that are forked. There are also 4 more that appear to be threads. Is that correct ?

There are quite a few threads in libvirtd, so lets go from the beginning. When libvirtd starts up, it is obviously single threaded. One of the first things we do is to create the server state object - virNetServerPtr. This object is what processes for incoming client connections, creating virNetServerClientPtr instances. The virNetServerPtr is also responsible for dispatching RPC messages. To allow maximum concurrency, the virNetServerPtr creates two pools of worker threads for handling RPC. One pool is the so called "high priority" threads pool, which handles RPC messages which are very fast and never block. The other thread pool handles all other RPC messages, including those which may take a long time. Each of these pools has an initial 5 threads, so we have 10 threads in total for RPC workers, but this number may grow over time - the thread limits are controlled in libvirtd.conf During initialization, we need to run the global driver init functions (eg qemudStartup and friends). This is done by spawning a new thread, which invokes virStateInit(), waits for it to complete and then exits. The initial process leader thread, invokes virNetServerRun() which runs the main event loop. So most of the time there the 10+ worker threads, plus the event loop thread. At times other threads may be invoked, eg during tunnelled migration the QEMU driver spawns an extra thread to handle I/O. The NWfilter code spawns other threads to handle certain operations. Also core dumps on crashing QEMU will spawn threads.

Now, the virEvents main polling loop is running in a forked thread, it has no access to worker threads that where created through virsh for example

virsh is a completely separate process from libvirtd.

Last, if a VM needs something to watch over the state of its network devices, a separate thread needs to be created to do that because the VMs libvirt daemon thread can't make use of the virEvent functions without creating a new poll-loop thread ?

Any event loop callback must be sure to complete in very fast time, since all I/O processsing is blocked while an event callback is running. So typically callbacks would restrict themselvs to reading/writing I/O on the triggering file descriptor, or some other small non-blocking piece of worker. If your event loop callback intends to invoke complex APis, which may in turn need to do I/O, then you must spawn a new thread so that you don't block the whole event loop. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On 12/29/2011 04:03 AM, D. Herrendoerfer wrote:

Sorry to come back on this.

[Caveat - I didn't write the threading model used in libvirt, so my answers are my interpretation of the code as I have come to understand it, but might not be 100% accurate]

How are the different threads of the libvirt daemon separated ? I understand that there are 2 separate threads running that are forked. There are also 4 more that appear to be threads. Is that correct ?

I'm not quite sure what you were asking, or which processes vs. threads you are viewing. Remember, in Linux, threads and processes share the same namespace of thread ids (that is, a process id in Linux is the thread-id of its primary thread), and the difference between a single-threaded app and a multi-threaded app is whether the process owns more than one thread id. Calling fork() creates a new process (and thus a new thread), and the process of starting libvirtd ends up calling fork() twice to get the daemonization right, so that may explain the "2 separate threads running that are forked". Once the daemon is up and running, it becomes multi-threaded, by spawning helper threads all within the same process, which would explain the "4 more that appear to be threads". Basically, libvirtd is set up in an event loop, where the main thread exists solely to monitor shutdown requests, and all other threads do round-robin servicing of RPC events that come in over the sockets.

Now, the virEvents main polling loop is running in a forked thread, it has no access to worker threads that where created through virsh for example Is that correct ?

Threads aren't forked, only processes are (but forking a new process creates a new thread, mainly the primary thread of the new process). Also, processes do not share memory, but threads within a process do, so a polling loop in a main thread _does_ have access to worker threads in the same process. It appears that you have some confusion in the terminology, which is in turn making it hard for me to know if I'm answering your questions. This question added in virsh, which adds another wrinkle into the picture - virsh is often capable of doing its job as a single-threaded application, which connects to the libvirtd socket to request libvirtd to do something with the qemu hypervisor. However, virsh is also designed for multi-threaded tasks - in particular, the ability to start a long-running task such as a migration, and cancel it midway, requires that virsh have a couple of threads, the primary thread that processes user commands delegates to a helper thread that interacts with libvirtd as well as listening for Ctrl-C events from the user or for migration completion events from libvirtd. This means that both libvirtd _and_ virsh are using an event loop - libvirtd to listen for incoming activity on the sockets as well as completion of those activities by the hypervisor, and virsh to listen for incoming activity from libvirtd completion. So in true code reuse fashion, we reuse the event loop in both processes, and you thus have to be careful which process you are asking about when discussing threads that are handling various events.

Last, if a VM needs something to watch over the state of its network devices, a separate thread needs to be created to do that because the VMs libvirt daemon thread can't make use of the virEvent functions without creating a new poll-loop thread ?

The event loop driver supports both file-based events, such as recognizing when traffic arrives on the public libvirtd socket which needs handling by one of the RPC worker threads, as well as time-based events. But it is not limited to the public libvirtd socket; the qemu hypervisor driver can register additional fd's to be watched, such as the monitor socket tied to the qemu running each domain. As long as you can come up with a scenario where an fd can track a change, then you can register that fd with the existing event loop. And don't forget that you can use pipes as a way to set up fds (after all, our SIGINT handler for Ctrl-C in virsh is nothing more than a pipe-to-self so that we have an fd to wake up the event loop).

Best regards,

D.Herrendoerfer <herrend at de dot ibm dot com > <d.herrendoerfer at herrendoerfer dot name>

On Dec 22, 2011, at 6:04 PM, Dirk Herrendoerfer wrote:

...

Hi all,

I'm trying to get libvirt to re-associate lost connections when a vepa connection is lost due to a switch error, or lldpad restart.

My take was to use the virtEvent infrastructure to poll for messages on a netlink socket and then restart the association if the message indicates that a link came back up.

I ran into a problem, that if I start the polling netlink event from the daemon thread I would get the file events an the netlink messages, but I cannot configure the event handler because the rpc client threads and the daemon do not share the same address space.

So it sounds like you are asking how to make a client, such as virsh, aware that an event was recognized in libvirtd. At this point, it boils down to an RPC event - whenever something in libvirtd fires that a client might usefully want to learn about, we can then map that to the RPC event mechanisms. Look at the virConnectDomainEventRegisterAny API, which is a means for setting up a callback function in the client that will be called when the event happens in libvirtd, by means of an RPC message from libvirtd to the client containing the details appropriate for the callback. Also, docs/internals/rpc.html describes the details behind RPC event calls; normally, things are driven by the client (client issues a call, server issues a response), but in the case of events, it is an asynchronous call where the server issues the event, and the client reacts to the event by calling any registered callback functions. I'm not sure if I'm on the right track for answering what you are really trying to do, but hopefully it helped, or someone else can chime in to correct anything I got wrong.

...

Is there a way to get file events in the VMs setup code so I can register a callback at VM initialization time to receive netlink messages and restart association if needed ?

Best regards,

D.Herrendoerfer <herrend at de dot ibm dot com > <d.herrendoerfer at herrendoerfer dot name>

-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

-- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org