Re: [libvirt-users] libvirt hang on domain attemp to create domain

On 20.06.2013 12:56, Alexandr wrote: > On 20.06.2013 10:44, Michal Privoznik wrote: >> On 20.06.2013 09:40, Александр wrote: >>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали: >>>> On 19.06.2013 21:24, Александр wrote: >>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik >>> >>> написал: >>>>>> On 18.06.2013 22:21, Александр wrote: >>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal >>>>>>> Privoznik >>>>>>> >>>>>>> написал: >>>>>>>> On 18.06.2013 07:14, Александр wrote: >>>>>>>>> good day, i am trying to create domain via virDomainDefineXML. >>>>>>>>> and my >>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon with >>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and >>>>>>>>> restarting >>>>>>>>> it solveing hang (i mean after restart i able to connect with >>>>>>>>> virt-manager), i using following domain xml >>>>>>>>> >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>>> <domain type="kvm"> >>>>>>>>> >>>>>>>>> <tittle>Automatically created domain with default >>>>>>>>> configuration</tittle> >>>>>>>>> <name>vm1</name> >>>>>>>>> <description>no description</description> >>>>>>>>> <os> >>>>>>>>> >>>>>>>>> <type>hvm</type> >>>>>>>>> >>>>>>>>> <boot dev="cdrom"/> >>>>>>>>> <boot dev="hd"/> >>>>>>>>> >>>>>>>>> </os> >>>>>>>>> >>>>>>>>> <vcpu>1</vcpu> >>>>>>>>> <memory unit="MB">512</memory> >>>>>>>>> <cpu model="host-model"> >>>>>>>>> >>>>>>>>> <topology sockets="1" cores="1" threads="1"/> >>>>>>>>> >>>>>>>>> </cpu> >>>>>>>>> >>>>>>>>> <clock offset="localtime"/> >>>>>>>>> <devices> >>>>>>>>> >>>>>>>>> <graphics type="vnc" autoport="yes" password="admin"/> >>>>>>>>> >>>>>>>>> <channel type="unix"> >>>>>>>>> >>>>>>>>> <source mode="bind" path="/var/run/vm1.sock"/> >>>>>>>>> >>>>>>>>> <target type="virtio" name="org.guest_agent.0"/> >>>>>>>>> >>>>>>>>> </channel> >>>>>>>>> >>>>>>>>> </devices> >>>>>>>>> >>>>>>>>> </domain> >>>>>>>>> >>>>>>>>> i have this calls on stack >>>>>>>>> >>>>>>>>> 0 0x00007ffff48c1750 __poll_nocancel >>>>>>>>> 1 0x00007ffff649e5ca virNetClientIOEventLoop >>>>>>>> >>>>>>>> This is just probably a client waiting for the corresponding >>>>>>>> libvirtd >>>>>>>> to >>>>>>>> provide a reply message. Are you able to get libvirtd debug logs >>>>>>>> so we >>>>>>>> can see what's going on? >>>>>>>> >>>>>>>> http://wiki.libvirt.org/page/DebugLogs >>>>>>>> >>>>>>>> BTW, what's the libvirt version you are running? >>>>>>>> >>>>>>>> Michal >>>>>>>> >>>>>>>>> 2 0x00007ffff649ed96 virNetClientSendInternal >>>>>>>>> 3 0x00007ffff64a0443 virNetClientSendWithReply >>>>>>>>> 4 0x00007ffff64a0daa virNetClientProgramCall >>>>>>>>> 5 0x00007ffff6478566 callFull.isra.2 >>>>>>>>> 6 0x00007ffff6487fa2 remoteDomainDefineXML >>>>>>>>> 7 0x00007ffff6449d2f virDomainDefineXML >>>>>>>>> 8 0x0000000000426941 virt_domain::create >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 >>>>>>>>> 9 0x000000000040c1a9 main >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 >>>>>>>>> >>>>>>>>> what i am doing wrong ? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> libvirt-users mailing list >>>>>>>>> libvirt-users(a)redhat.com >>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users >>>>>>> >>>>>>> i have libvirt version 1.0.6, debug log in attachment. >>>>>> >>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is >>>>>> stuck. Can you attach to the libvirtd while it's being stuck and >>>>>> run 't >>>>>> a a bt' and post the result? I wonder if we have a deadlock >>>>>> somewhere. >>>>>> >>>>>> Michal >>>>> >>>>> if i correctly understand you, you need debugger backtrace after >>>>> libvirtd >>>>> stuck ?, if so, here is bt from gdb >>>> >>>> You understand correctly. However, simple 'bt' doesn't show what's >>>> going >>>> on in other threads - libvirtd is multi threaded. >>>> >>>>> (gdb) bt >>>>> #0 0x00007f00c1ae876d in poll () from /lib64/libc.so.6 >>>>> #1 0x00007f00c203cfcb in virEventPollRunOnce () from >>>>> /usr/lib64/libvirt.so.0 #2 0x00007f00c203ba4d in >>>>> virEventRunDefaultImpl >>>>> () from >>>>> /usr/lib64/libvirt.so.0 >>>>> #3 0x00007f00c214a5cd in virNetServerRun () from >>>>> /usr/lib64/libvirt.so.0 >>>>> #4 0x00007f00c2aea08a in main () >>>> >>>> So all we see here is the libvirtd event loop. That's why we need 't >>>> a a >>>> bt' which is 'thread apply all backtrace' which will run 'bt' over all >>>> threads. >>>> >>>> Michal >>> >>> how can i make it ? >> >> The very same way as you did 'bt'. Just instead of typing 'bt' you'll >> type 't a a bt'. >> >> Michal > > here is backtrace from all threads: > > (gdb) t a a bt > > Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476582b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476582b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)): > #0 0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0 > #2 0x00007fdd944b7caa in pthread_mutex_lock () from > /lib64/libpthread.so.0 Bingo! So we've a deadlock here. Somebody has forgotten to unlock a domain. So as we go over the domain list to check for name (if it exists already or not), we deadlock. The process goes like this: 1) if adding a domain the the internal list (I should write rather hash table where UUID is the key, and the pointer to the internal structure representing domain is the value), we search for UUID. Since this is the key, we don't have to lock anything (besides the hash table itself). 2) Okay, no matching UUID. But what about domain name? We go through the table again, over each element, and: 2a) lock the element 2b) compare the names 2c) unlock the element And since a domain withing the table is locked (and all none of the threads is using it) we deadlock in step 2a) as there's no one to come and unlock the domain. Sigh. Hopefully, I'll have some time to dig into the log you've provided and get more precise info. > #3 0x00007fdd94773861 in virDomainObjListSearchName () from > /usr/lib64/libvirt.so.0 > #4 0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0 > #5 0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from > /usr/lib64/libvirt.so.0 > #6 0x00007fdd9478cd47 in virDomainObjListAdd () from > /usr/lib64/libvirt.so.0 > #7 0x00007fdd875a311c in qemuDomainDefineXML () from > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > #8 0x00007fdd947ead2f in virDomainDefineXML () from > /usr/lib64/libvirt.so.0 > #9 0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper () > #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from > /usr/lib64/libvirt.so.0 > #11 0x00007fdd94844068 in virNetServerHandleJob () from > /usr/lib64/libvirt.so.0 > #12 0x00007fdd94765795 in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476582b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476582b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476584b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476584b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > ---Type <return> to continue, or q <return> to quit--- > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476584b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476584b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 2 (Thread 0x7fdd89379700 (LWP 20330)): > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > #2 0x00007fdd9476584b in virThreadPoolWorker () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > Thread 1 (Thread 0x7fdd95166740 (LWP 20320)): > #0 0x00007fdd941e376d in poll () from /lib64/libc.so.6 > #1 0x00007fdd94737fcb in virEventPollRunOnce () from > /usr/lib64/libvirt.so.0 > #2 0x00007fdd94736a4d in virEventRunDefaultImpl () from > /usr/lib64/libvirt.so.0 > #3 0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0 > #4 0x00007fdd951e508a in main () > > and also from my program if it important: > > 0 0x00007ffff48c1750 __poll_nocancel > 1 0x00007ffff649e5ca virNetClientIOEventLoop > 2 0x00007ffff649ed96 virNetClientSendInternal > 3 0x00007ffff64a0443 virNetClientSendWithReply > 4 0x00007ffff64a0daa virNetClientProgramCall > 5 0x00007ffff6478566 callFull.isra.2 > 6 0x00007ffff6487fa2 remoteDomainDefineXML > 7 0x00007ffff6449d2f virDomainDefineXML > 8 0x0000000000426941 virt_domain::create > /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 > 9 0x000000000040c1a9 main > /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 Michal