2:04 p.m.
Hi,
i have found the time and tested Xen-4.4.1 with libvirt-1.2.6 and the problem with the
threads are disappered. But now i see after every restore of an xen-vm (windows-xp hvm) an
openfile-handle (via lsof -p `pgrep libvirtd`) to an file in /var/lib/xen with the name
qemu-resume.<vm-id> and what me real make confusion is that the entry is marked as
deleted.If i look in the directory, the file is really no more their, but libvirt have
still have an handle to that none existing file!?!
I know that "Xen" create such a file before they startet the vm, and i also know
that xen delete that file , but i don´t understand while it is listed under the openfiles
of libvirtd after the vm is started/destroyed???
all the best
max
----------------ursprüngliche Nachricht-----------------
Von: "web2" ustermann78(a)web.de
An: "Jim Fehlig" jfehlig(a)suse.com , "web2" ustermann78(a)web.de
Kopie: "Michal Privoznik" mprivozn(a)redhat.com , "libvirt-users
redhat.com"
, "libvirt-list redhat.com"
Datum: Thu, 2 Oct 2014 07:49:03 +0000
-------------------------------------------------
----------------ursprüngliche Nachricht-----------------
Von: "Jim Fehlig" jfehlig(a)suse.com
An: "web2" ustermann78(a)web.de
Kopie: "Michal Privoznik" mprivozn(a)redhat.com , "libvirt-users
redhat.com"
, "libvirt-list redhat.com"
Datum: Wed, 01 Oct 2014 21:38:05 -0600
-------------------------------------------------
> web2 wrote:
>> Hi
>>
>> ----------------ursprüngliche Nachricht-----------------
>> Von: "Michal Privoznik" mprivozn(a)redhat.com
>> An: "web2" ustermann78(a)web.de , "libvirt-users redhat.com"
>> , "libvirt-list redhat.com"
>> Datum: Wed, 01 Oct 2014 18:12:45 +0200
>> -------------------------------------------------
>>
>>
>>
>>> On 01.10.2014 10:31, web2 wrote:
>>>
>>>> Hello,
>>>>
>>>> sorry for my later answer.
>>>>
>>>> so, after i started libvirtd (no vm´s running) and attach gdb i get
>>>> the
>>>> following threads
>>>>
>>>> (gdb) info thread
>>>> Id Target Id Frame
>>>> 11 Thread 0x7f18fef4e700 (LWP 20695) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 10 Thread 0x7f18fe74d700 (LWP 20696) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 9 Thread 0x7f18fdf4c700 (LWP 20697) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 8 Thread 0x7f18fd74b700 (LWP 20698) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 7 Thread 0x7f18fcf4a700 (LWP 20699) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 6 Thread 0x7f18fc749700 (LWP 20700) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 5 Thread 0x7f18fbf48700 (LWP 20701) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 4 Thread 0x7f18fb747700 (LWP 20702) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 3 Thread 0x7f18faf46700 (LWP 20703) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 2 Thread 0x7f18fa745700 (LWP 20704) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> * 1 Thread 0x7f190892f840 (LWP 20694) "libvirtd"
>>>> 0x00007f190677d7cd in
>>>> poll () at ../sysdeps/unix/syscall-template.S:81
>>>>
>>>> if i restore an persistent domain, i see the following in gdb:
>>>>
>>>> Detaching after fork from child process 20880.
>>>> Detaching after fork from child process 20882.
>>>> [New Thread 0x7f190893d700 (LWP 20883)]
>>>> Detaching after fork from child process 20890.
>>>> Detaching after fork from child process 20906.
>>>>
>>>> (gdb) info thread
>>>> Id Target Id Frame
>>>> 12 Thread 0x7f190893d700 (LWP 20883) "libvirtd"
>>>> 0x00007f1906e6684d in
>>>> read () at ../sysdeps/unix/syscall-template.S:81
>>>> 11 Thread 0x7f18fef4e700 (LWP 20695) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 10 Thread 0x7f18fe74d700 (LWP 20696) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 9 Thread 0x7f18fdf4c700 (LWP 20697) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 8 Thread 0x7f18fd74b700 (LWP 20698) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 7 Thread 0x7f18fcf4a700 (LWP 20699) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 6 Thread 0x7f18fc749700 (LWP 20700) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 5 Thread 0x7f18fbf48700 (LWP 20701) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 4 Thread 0x7f18fb747700 (LWP 20702) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 3 Thread 0x7f18faf46700 (LWP 20703) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 2 Thread 0x7f18fa745700 (LWP 20704) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> * 1 Thread 0x7f190892f840 (LWP 20694) "libvirtd"
>>>> 0x00007f190677d7cd in
>>>> poll () at ../sysdeps/unix/syscall-template.S:81
>>>>
>>>> if i now destroy this vm i get the following:
>>>>
>>>>
>>>> (gdb) info thread
>>>> Id Target Id Frame
>>>> 12 Thread 0x7f190893d700 (LWP 20883) "libvirtd"
>>>> 0x00007f1906e6684d in
>>>> read () at ../sysdeps/unix/syscall-template.S:81
>>>> 11 Thread 0x7f18fef4e700 (LWP 20695) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 10 Thread 0x7f18fe74d700 (LWP 20696) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 9 Thread 0x7f18fdf4c700 (LWP 20697) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 8 Thread 0x7f18fd74b700 (LWP 20698) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 7 Thread 0x7f18fcf4a700 (LWP 20699) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 6 Thread 0x7f18fc749700 (LWP 20700) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 5 Thread 0x7f18fbf48700 (LWP 20701) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 4 Thread 0x7f18fb747700 (LWP 20702) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 3 Thread 0x7f18faf46700 (LWP 20703) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> 2 Thread 0x7f18fa745700 (LWP 20704) "libvirtd"
>>>> pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>>
>>>>
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:1
>>>> 85
>>>>
>>>> * 1 Thread 0x7f190892f840 (LWP 20694) "libvirtd"
>>>> 0x00007f190677d7cd in
>>>> poll () at ../sysdeps/unix/syscall-template.S:81
>>>>
>>>> attaching thread Id 12 and bt:
>>>> (gdb) thread 12
>>>> [Switching to thread 12 (Thread 0x7f190893d700 (LWP 20883))]
>>>> #0 0x00007f1906e6684d in read () at
>>>> ../sysdeps/unix/syscall-template.S:81
>>>> 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
>>>> (gdb) bt
>>>> #0 0x00007f1906e6684d in read () at
>>>> ../sysdeps/unix/syscall-template.S:81
>>>> #1 0x00007f18f8c415be in read (__nbytes=16,
>>>> __buf=0x7f18d4000c50,
>>>> __fd=26) at /usr/include/bits/unistd.h:44
>>>> #2 read_all (fd=26, data=data@entry =0x7f18d4000c50,
>>>> len=len@entry =16,
>>>> nonblocking=nonblocking@entry =0) at xs.c:374
>>>> #3 0x00007f18f8c41675 in read_message (h=h@entry
>>>> =0x7f18e8001070,
>>>> nonblocking=nonblocking@entry =0) at xs.c:1139
>>>> #4 0x00007f18f8c41e90 in read_thread (arg=0x7f18e8001070) at
>>>> xs.c:1211
>>>> #5 0x00007f1906e5ff35 in start_thread (arg=0x7f190893d700) at
>>>>
>>>> pthread_create.c:309
>>>> #6 0x00007f1906787c3d in clone () at
>>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>>>>
>>>> and then bt full:
>>>>
>>>> (gdb) bt full
>>>> #0 0x00007f1906e6684d in read () at
>>>> ../sysdeps/unix/syscall-template.S:81
>>>> No locals.
>>>> #1 0x00007f18f8c415be in read (__nbytes=16,
>>>> __buf=0x7f18d4000c50,
>>>> __fd=26) at /usr/include/bits/unistd.h:44
>>>> No locals.
>>>> #2 read_all (fd=26, data=data@entry =0x7f18d4000c50,
>>>> len=len@entry =16,
>>>> nonblocking=nonblocking@entry =0) at xs.c:374
>>>> done = <optimized out>
>>>> #3 0x00007f18f8c41675 in read_message (h=h@entry
>>>> =0x7f18e8001070,
>>>> nonblocking=nonblocking@entry =0) at xs.c:1139
>>>> __clframe = {__cancel_routine = <optimized out>, __cancel_arg =
>>>>
>>>> 0x7f18d4000c40, __do_it = 1, __cancel_type = <optimized out>}
>>>> msg = 0x7f18d4000c40
>>>> body = 0x0
>>>> saved_errno = 0
>>>> ret = -1
>>>> #4 0x00007f18f8c41e90 in read_thread (arg=0x7f18e8001070) at
>>>> xs.c:1211
>>>> h = 0x7f18e8001070
>>>> fd = <optimized out>
>>>>
>>> This stack doesn't come from libvirt. I suspect it comes from a library
>>>
>>> that libvirtd is linked with.
>
> Yep, a thread created by libxl.
>
>>> Maybe libxl? What's the connection uri
>>> you're using?
>>>
>>
>> i use xen:///. as i said early, i do tests with xen xm´s. xend is disabled, so
>> i
>> use the libxl.
>>
>
> Earlier you mentioned using libvirt 1.1.3 for your Xen hosts. Have you
> tried the freshly released 1.2.9? I didn't see the problem with libvirt
> git master and Xen 4.4.1.
I use libvirt 1.1.3.5 and Xen 4.3.3 for the tests. I will try a newer version of Xen
and then a newer version of libvirt.
thanks
max
> Regards,
> Jim
>
>
--