On 20.06.2013 10:44, Michal Privoznik wrote: > On 20.06.2013 09:40, Александр wrote: >> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали: >>> On 19.06.2013 21:24, Александр wrote: >>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik >> написал: >>>>> On 18.06.2013 22:21, Александр wrote: >>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal >>>>>> Privoznik >>>>>> >>>>>> написал: >>>>>>> On 18.06.2013 07:14, Александр wrote: >>>>>>>> good day, i am trying to create domain via virDomainDefineXML. >>>>>>>> and my >>>>>>>> program hanging, also i failed to caonnect to libvirt daemon with >>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and >>>>>>>> restarting >>>>>>>> it solveing hang (i mean after restart i able to connect with >>>>>>>> virt-manager), i using following domain xml >>>>>>>> >>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>> <domain type="kvm"> >>>>>>>> >>>>>>>> <tittle>Automatically created domain with default >>>>>>>> configuration</tittle> >>>>>>>> <name>vm1</name> >>>>>>>> <description>no description</description> >>>>>>>> <os> >>>>>>>> <type>hvm</type> >>>>>>>> <boot dev="cdrom"/> >>>>>>>> <boot dev="hd"/> >>>>>>>> </os> >>>>>>>> <vcpu>1</vcpu> >>>>>>>> <memory unit="MB">512</memory> >>>>>>>> <cpu model="host-model"> >>>>>>>> <topology sockets="1" cores="1" threads="1"/> >>>>>>>> </cpu> >>>>>>>> <clock offset="localtime"/> >>>>>>>> <devices> >>>>>>>> <graphics type="vnc" autoport="yes" password="admin"/> >>>>>>>> <channel type="unix"> >>>>>>>> <source mode="bind" path="/var/run/vm1.sock"/> >>>>>>>> <target type="virtio" name="org.guest_agent.0"/> >>>>>>>> </channel> >>>>>>>> </devices> >>>>>>>> >>>>>>>> </domain> >>>>>>>> >>>>>>>> i have this calls on stack >>>>>>>> >>>>>>>> 0 0x00007ffff48c1750 __poll_nocancel >>>>>>>> 1 0x00007ffff649e5ca virNetClientIOEventLoop >>>>>>> This is just probably a client waiting for the corresponding >>>>>>> libvirtd >>>>>>> to >>>>>>> provide a reply message. Are you able to get libvirtd debug logs >>>>>>> so we >>>>>>> can see what's going on? >>>>>>> >>>>>>> http://wiki.libvirt.org/page/DebugLogs >>>>>>> >>>>>>> BTW, what's the libvirt version you are running? >>>>>>> >>>>>>> Michal >>>>>>> >>>>>>>> 2 0x00007ffff649ed96 virNetClientSendInternal >>>>>>>> 3 0x00007ffff64a0443 virNetClientSendWithReply >>>>>>>> 4 0x00007ffff64a0daa virNetClientProgramCall >>>>>>>> 5 0x00007ffff6478566 callFull.isra.2 >>>>>>>> 6 0x00007ffff6487fa2 remoteDomainDefineXML >>>>>>>> 7 0x00007ffff6449d2f virDomainDefineXML >>>>>>>> 8 0x0000000000426941 virt_domain::create >>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 >>>>>>>> 9 0x000000000040c1a9 main >>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 >>>>>>>> >>>>>>>> what i am doing wrong ? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> libvirt-users mailing list >>>>>>>> libvirt-users(a)redhat.com >>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users >>>>>> i have libvirt version 1.0.6, debug log in attachment. >>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is >>>>> stuck. Can you attach to the libvirtd while it's being stuck and >>>>> run 't >>>>> a a bt' and post the result? I wonder if we have a deadlock >>>>> somewhere. >>>>> >>>>> Michal >>>> if i correctly understand you, you need debugger backtrace after >>>> libvirtd >>>> stuck ?, if so, here is bt from gdb >>> You understand correctly. However, simple 'bt' doesn't show what's >>> going >>> on in other threads - libvirtd is multi threaded. >>> >>>> (gdb) bt >>>> #0 0x00007f00c1ae876d in poll () from /lib64/libc.so.6 >>>> #1 0x00007f00c203cfcb in virEventPollRunOnce () from >>>> /usr/lib64/libvirt.so.0 #2 0x00007f00c203ba4d in >>>> virEventRunDefaultImpl >>>> () from >>>> /usr/lib64/libvirt.so.0 >>>> #3 0x00007f00c214a5cd in virNetServerRun () from >>>> /usr/lib64/libvirt.so.0 >>>> #4 0x00007f00c2aea08a in main () >>> So all we see here is the libvirtd event loop. That's why we need 't >>> a a >>> bt' which is 'thread apply all backtrace' which will run 'bt' over all >>> threads. >>> >>> Michal >> how can i make it ? >> > The very same way as you did 'bt'. Just instead of typing 'bt' you'll > type 't a a bt'. > > Michal here is backtrace from all threads: (gdb) t a a bt Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476582b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476582b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)): #0 0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0 #2 0x00007fdd944b7caa in pthread_mutex_lock () from /lib64/libpthread.so.0

#3 0x00007fdd94773861 in virDomainObjListSearchName () from /usr/lib64/libvirt.so.0 #4 0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0 #5 0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from /usr/lib64/libvirt.so.0 #6 0x00007fdd9478cd47 in virDomainObjListAdd () from /usr/lib64/libvirt.so.0 #7 0x00007fdd875a311c in qemuDomainDefineXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007fdd947ead2f in virDomainDefineXML () from /usr/lib64/libvirt.so.0 #9 0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper () #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0 #11 0x00007fdd94844068 in virNetServerHandleJob () from /usr/lib64/libvirt.so.0 #12 0x00007fdd94765795 in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476582b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476582b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476584b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476584b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476584b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476584b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fdd89379700 (LWP 20330)): #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fdd9476584b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fdd95166740 (LWP 20320)): #0 0x00007fdd941e376d in poll () from /lib64/libc.so.6 #1 0x00007fdd94737fcb in virEventPollRunOnce () from /usr/lib64/libvirt.so.0 #2 0x00007fdd94736a4d in virEventRunDefaultImpl () from /usr/lib64/libvirt.so.0 #3 0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0 #4 0x00007fdd951e508a in main () and also from my program if it important: 0 0x00007ffff48c1750 __poll_nocancel 1 0x00007ffff649e5ca virNetClientIOEventLoop 2 0x00007ffff649ed96 virNetClientSendInternal 3 0x00007ffff64a0443 virNetClientSendWithReply 4 0x00007ffff64a0daa virNetClientProgramCall 5 0x00007ffff6478566 callFull.isra.2 6 0x00007ffff6487fa2 remoteDomainDefineXML 7 0x00007ffff6449d2f virDomainDefineXML 8 0x0000000000426941 virt_domain::create /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 9 0x000000000040c1a9 main /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40

В письме от Четверг, 20-июн-2013 13:25:06 пользователь Michal Privoznik написал: > On 20.06.2013 12:56, Alexandr wrote: > > On 20.06.2013 10:44, Michal Privoznik wrote: > >> On 20.06.2013 09:40, Александр wrote: > >>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали: > >>>> On 19.06.2013 21:24, Александр wrote: > >>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal > >>>>> Privoznik > >>> > >>> написал: > >>>>>> On 18.06.2013 22:21, Александр wrote: > >>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal > >>>>>>> Privoznik > >>>>>>> > >>>>>>> написал: > >>>>>>>> On 18.06.2013 07:14, Александр wrote: > >>>>>>>>> good day, i am trying to create domain via virDomainDefineXML. > >>>>>>>>> and my > >>>>>>>>> program hanging, also i failed to caonnect to libvirt daemon > >>>>>>>>> with > >>>>>>>>> virt-manager after this, only killing libvirtd via SIGKILL and > >>>>>>>>> restarting > >>>>>>>>> it solveing hang (i mean after restart i able to connect with > >>>>>>>>> virt-manager), i using following domain xml > >>>>>>>>> > >>>>>>>>> <?xml version="1.0" encoding="UTF-8"?> > >>>>>>>>> <domain type="kvm"> > >>>>>>>>> > >>>>>>>>> <tittle>Automatically created domain with default > >>>>>>>>> configuration</tittle> > >>>>>>>>> <name>vm1</name> > >>>>>>>>> <description>no description</description> > >>>>>>>>> <os> > >>>>>>>>> > >>>>>>>>> <type>hvm</type> > >>>>>>>>> > >>>>>>>>> <boot dev="cdrom"/> > >>>>>>>>> <boot dev="hd"/> > >>>>>>>>> > >>>>>>>>> </os> > >>>>>>>>> > >>>>>>>>> <vcpu>1</vcpu> > >>>>>>>>> <memory unit="MB">512</memory> > >>>>>>>>> <cpu model="host-model"> > >>>>>>>>> > >>>>>>>>> <topology sockets="1" cores="1" threads="1"/> > >>>>>>>>> > >>>>>>>>> </cpu> > >>>>>>>>> > >>>>>>>>> <clock offset="localtime"/> > >>>>>>>>> <devices> > >>>>>>>>> > >>>>>>>>> <graphics type="vnc" autoport="yes" password="admin"/> > >>>>>>>>> > >>>>>>>>> <channel type="unix"> > >>>>>>>>> > >>>>>>>>> <source mode="bind" path="/var/run/vm1.sock"/> > >>>>>>>>> > >>>>>>>>> <target type="virtio" name="org.guest_agent.0"/> > >>>>>>>>> > >>>>>>>>> </channel> > >>>>>>>>> > >>>>>>>>> </devices> > >>>>>>>>> > >>>>>>>>> </domain> > >>>>>>>>> > >>>>>>>>> i have this calls on stack > >>>>>>>>> > >>>>>>>>> 0 0x00007ffff48c1750 __poll_nocancel > >>>>>>>>> 1 0x00007ffff649e5ca virNetClientIOEventLoop > >>>>>>>> > >>>>>>>> This is just probably a client waiting for the corresponding > >>>>>>>> libvirtd > >>>>>>>> to > >>>>>>>> provide a reply message. Are you able to get libvirtd debug logs > >>>>>>>> so we > >>>>>>>> can see what's going on? > >>>>>>>> > >>>>>>>> http://wiki.libvirt.org/page/DebugLogs > >>>>>>>> > >>>>>>>> BTW, what's the libvirt version you are running? > >>>>>>>> > >>>>>>>> Michal > >>>>>>>> > >>>>>>>>> 2 0x00007ffff649ed96 virNetClientSendInternal > >>>>>>>>> 3 0x00007ffff64a0443 virNetClientSendWithReply > >>>>>>>>> 4 0x00007ffff64a0daa virNetClientProgramCall > >>>>>>>>> 5 0x00007ffff6478566 callFull.isra.2 > >>>>>>>>> 6 0x00007ffff6487fa2 remoteDomainDefineXML > >>>>>>>>> 7 0x00007ffff6449d2f virDomainDefineXML > >>>>>>>>> 8 0x0000000000426941 virt_domain::create > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp > >>>>>>>>> 295 > >>>>>>>>> 9 0x000000000040c1a9 main > >>>>>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 > >>>>>>>>> > >>>>>>>>> what i am doing wrong ? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> libvirt-users mailing list > >>>>>>>>> libvirt-users(a)redhat.com > >>>>>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users > >>>>>>> > >>>>>>> i have libvirt version 1.0.6, debug log in attachment. > >>>>>> > >>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon > >>>>>> is > >>>>>> stuck. Can you attach to the libvirtd while it's being stuck and > >>>>>> run 't > >>>>>> a a bt' and post the result? I wonder if we have a deadlock > >>>>>> somewhere. > >>>>>> > >>>>>> Michal > >>>>> > >>>>> if i correctly understand you, you need debugger backtrace after > >>>>> libvirtd > >>>>> stuck ?, if so, here is bt from gdb > >>>> > >>>> You understand correctly. However, simple 'bt' doesn't show what's > >>>> going > >>>> on in other threads - libvirtd is multi threaded. > >>>> > >>>>> (gdb) bt > >>>>> #0 0x00007f00c1ae876d in poll () from /lib64/libc.so.6 > >>>>> #1 0x00007f00c203cfcb in virEventPollRunOnce () from > >>>>> /usr/lib64/libvirt.so.0 #2 0x00007f00c203ba4d in > >>>>> virEventRunDefaultImpl > >>>>> () from > >>>>> /usr/lib64/libvirt.so.0 > >>>>> #3 0x00007f00c214a5cd in virNetServerRun () from > >>>>> /usr/lib64/libvirt.so.0 > >>>>> #4 0x00007f00c2aea08a in main () > >>>> > >>>> So all we see here is the libvirtd event loop. That's why we need 't > >>>> a a > >>>> bt' which is 'thread apply all backtrace' which will run 'bt' over > >>>> all > >>>> threads. > >>>> > >>>> Michal > >>> > >>> how can i make it ? > >> > >> The very same way as you did 'bt'. Just instead of typing 'bt' you'll > >> type 't a a bt'. > >> > >> Michal > > > > here is backtrace from all threads: > > > > (gdb) t a a bt > > > > Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)): > > #0 0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0 > > #1 0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0 > > #2 0x00007fdd944b7caa in pthread_mutex_lock () from > > /lib64/libpthread.so.0 > > Bingo! So we've a deadlock here. Somebody has forgotten to unlock a > domain. So as we go over the domain list to check for name (if it exists > already or not), we deadlock. The process goes like this: > > 1) if adding a domain the the internal list (I should write rather hash > table where UUID is the key, and the pointer to the internal structure > representing domain is the value), we search for UUID. Since this is the > key, we don't have to lock anything (besides the hash table itself). > > 2) Okay, no matching UUID. But what about domain name? We go through the > table again, over each element, and: > 2a) lock the element > 2b) compare the names > 2c) unlock the element > > And since a domain withing the table is locked (and all none of the > threads is using it) we deadlock in step 2a) as there's no one to come > and unlock the domain. Sigh. Hopefully, I'll have some time to dig into > the log you've provided and get more precise info. > > > #3 0x00007fdd94773861 in virDomainObjListSearchName () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0 > > #5 0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from > > /usr/lib64/libvirt.so.0 > > #6 0x00007fdd9478cd47 in virDomainObjListAdd () from > > /usr/lib64/libvirt.so.0 > > #7 0x00007fdd875a311c in qemuDomainDefineXML () from > > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > > #8 0x00007fdd947ead2f in virDomainDefineXML () from > > /usr/lib64/libvirt.so.0 > > #9 0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper () > > #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from > > /usr/lib64/libvirt.so.0 > > #11 0x00007fdd94844068 in virNetServerHandleJob () from > > /usr/lib64/libvirt.so.0 > > #12 0x00007fdd94765795 in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #13 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476582b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > ---Type <return> to continue, or q <return> to quit--- > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 2 (Thread 0x7fdd89379700 (LWP 20330)): > > #0 0x00007fdd944b990c in pthread_cond_wait@(a)GLIBC_2.3.2 () from > > /lib64/libpthread.so.0 > > #1 0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0 > > #2 0x00007fdd9476584b in virThreadPoolWorker () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd94764dfe in virThreadHelper () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00007fdd941ed04d in clone () from /lib64/libc.so.6 > > > > Thread 1 (Thread 0x7fdd95166740 (LWP 20320)): > > #0 0x00007fdd941e376d in poll () from /lib64/libc.so.6 > > #1 0x00007fdd94737fcb in virEventPollRunOnce () from > > /usr/lib64/libvirt.so.0 > > #2 0x00007fdd94736a4d in virEventRunDefaultImpl () from > > /usr/lib64/libvirt.so.0 > > #3 0x00007fdd948455cd in virNetServerRun () from > > /usr/lib64/libvirt.so.0 > > #4 0x00007fdd951e508a in main () > > > > and also from my program if it important: > > > > 0 0x00007ffff48c1750 __poll_nocancel > > 1 0x00007ffff649e5ca virNetClientIOEventLoop > > 2 0x00007ffff649ed96 virNetClientSendInternal > > 3 0x00007ffff64a0443 virNetClientSendWithReply > > 4 0x00007ffff64a0daa virNetClientProgramCall > > 5 0x00007ffff6478566 callFull.isra.2 > > 6 0x00007ffff6487fa2 remoteDomainDefineXML > > 7 0x00007ffff6449d2f virDomainDefineXML > > 8 0x0000000000426941 virt_domain::create > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 295 > > 9 0x000000000040c1a9 main > > /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 40 > > Michal thx for quick reply, i will wait.