5:22 a.m.
Dear list,
I'm trying to package libvirt 0.8.1 for our distribution, Pardus 2009.2.
libvirt is installed perfectly normal, and libvirtd runs OK when I start it
in a console using root account.
However, when I start libvirtd as a service, with the same parameters,
through the normal service startup functions, it segfaults.
The services in Pardus 2009.2 are started using a management backend which
works with python and service start/stop scripts are python based.
For libvirt, it's the following:
http://svn.pardus.org.tr/pardus/playground/ozan/libvirt/comar/service.py
Whatever I did, I couldn't find why libvirt is crashing. It works normal
when I run it from console with exactly the same parameters. Here's an
earlier syslog section ending with the crash:
May 21 17:34:31 voyager kernel: [148326.330658] Bridge firewalling
registered
May 21 17:34:31 voyager kernel: [148326.626654] virbr0: starting userspace
STP failed, starting kernel STP
May 21 17:34:31 voyager avahi-daemon[559]: Joining mDNS multicast group on
interface virbr0.IPv4 with address 192.168.122.1.
May 21 17:34:31 voyager avahi-daemon[559]: New relevant interface
virbr0.IPv4 for mDNS.
May 21 17:34:31 voyager avahi-daemon[559]: Registering new address record
for 192.168.122.1 on virbr0.IPv4.
May 21 17:34:32 voyager kernel: [148327.158860] ip_tables: (C) 2000-2006
Netfilter Core Team
May 21 17:34:32 voyager kernel: [148327.359921] nf_conntrack version 0.5.0
(16384 buckets, 65536 max)
May 21 17:34:32 voyager kernel: [148327.360680] CONFIG_NF_CT_ACCT is
deprecated and will be removed soon. Please use
May 21 17:34:32 voyager kernel: [148327.360682] nf_conntrack.acct=1 kernel
parameter, acct=1 nf_conntrack module option or
May 21 17:34:32 voyager kernel: [148327.360683] sysctl
net.netfilter.nf_conntrack_acct=1 to enable it.
May 21 17:34:33 voyager dnsmasq[13918]: started, version 2.47 cachesize
150
May 21 17:34:33 voyager dnsmasq[13918]: compile time options: IPv6
GNU-getopt DBus no-I18N TFTP
May 21 17:34:33 voyager dnsmasq[13918]: DHCP, IP range 192.168.122.2 --
192.168.122.254, lease time 1h
May 21 17:34:33 voyager dnsmasq[13918]: reading
/etc/resolv.conf
May 21 17:34:33 voyager dnsmasq[13918]: using nameserver
193.140.100.215#53
May 21 17:34:33 voyager dnsmasq[13918]: using nameserver
193.140.100.210#53
May 21 17:34:33 voyager dnsmasq[13918]: using nameserver 192.168.1.1#53
May 21 17:34:33 voyager dnsmasq[13918]: read /etc/hosts - 7 addresses
May 21 17:34:33 voyager kernel: [148328.620605] libvirtd[13826]: segfault at
0 ip b74aa8f3 sp bfcd24fc error 4 in libc-2.9.so[b7431000+161000]
Thanks for any insight,
Best Regards,
--
Emre Erenoglu
6:05 a.m.
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
Dear list,
I'm trying to package libvirt 0.8.1 for our distribution, Pardus 2009.2.
libvirt is installed perfectly normal, and libvirtd runs OK when I start it
in a console using root account.
However, when I start libvirtd as a service, with the same parameters,
through the normal service startup functions, it segfaults.
The services in Pardus 2009.2 are started using a management backend which
works with python and service start/stop scripts are python based.
For libvirt, it's the following:
http://svn.pardus.org.tr/pardus/playground/ozan/libvirt/comar/service.py
Whatever I did, I couldn't find why libvirt is crashing. It works normal
when I run it from console with exactly the same parameters. Here's an
earlier syslog section ending with the crash:
There are some things to consider:
- Did you use the exact same commandline as the initscript when
testing manually?
- Did you make sure to use the same environment variable configuration
when starting libvirtd manually, compared to the initscript?
Could you provide a GDB backtrace of the segfault? The syslog entry
only says that it crashed in libc, that's not enough information to
debug the segfault.
Matthias
8:10 a.m.
On Thu, Jun 10, 2010 at 2:05 PM, Matthias Bolte <
matthias.bolte(a)googlemail.com> wrote:
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> Dear list,
>
> I'm trying to package libvirt 0.8.1 for our distribution, Pardus 2009.2.
> libvirt is installed perfectly normal, and libvirtd runs OK when I start
it
> in a console using root account.
>
> However, when I start libvirtd as a service, with the same parameters,
> through the normal service startup functions, it segfaults.
>
> The services in Pardus 2009.2 are started using a management backend
which
> works with python and service start/stop scripts are python based.
>
> For libvirt, it's the following:
> http://svn.pardus.org.tr/pardus/playground/ozan/libvirt/comar/service.py
>
> Whatever I did, I couldn't find why libvirt is crashing. It works normal
> when I run it from console with exactly the same parameters. Here's an
> earlier syslog section ending with the crash:
>
There are some things to consider:
- Did you use the exact same commandline as the initscript when
testing manually?
Yes. In fact, the only parameter passed is the --daemon parameter with
current configuration.
- Did you make sure to use the same environment variable
configuration when
starting libvirtd manually, compared to the initscript?
Here's the environment of the root user, I will try to find out the
environment of the service script:
MANPATH=/usr/local/share/man:/usr/share/man:/opt/sun-jre/man:/usr/kde/4/share/man
HOSTNAME=EMRE
SHELL=/bin/bash
TERM=linux
XDG_SESSION_COOKIE=3d6ade2bb28141896f3212d64bf41670-1276174999.886063-1263776093
HUSHLOGIN=FALSE
LC_ALL=en_US.UTF-8
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.pisi=01;33:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.ogv=01\:35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
GUILE_LOAD_PATH=/usr/share/guile/1.8
MC_ENV=/usr/share/mc/bin/mc.sh
PAGER=/usr/bin/less
CONFIG_PROTECT_MASK=/etc/texmf/web2c /etc/texmf/language.dat.d
/etc/texmf/language.def.d /etc/texmf/updmap.d
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/opt/sun-jre/bin:/usr/kde/4/sbin:/usr/kde/4/bin
PWD=/root
JAVA_HOME=/opt/sun-jre
EDITOR=/bin/nano
LESSCOLOR=yes
LANG=en_US.UTF-8
PYTHONSTARTUP=/etc/pythonstart
PS1=\[\033[1;31m\]\h \[\033[1;34m\]\W \$ \[\033[00m\]
SHLVL=1
HOME=/root
LD_BIND_DIRECT=1
LESS=-R -M --shift 5
LOGNAME=root
CVS_RSH=ssh
XDG_DATA_DIRS=/usr/kde/4/share:/usr/share
PKG_CONFIG_PATH=/usr/kde/4/lib/pkgconfig:/usr/qt/4/lib/pkgconfig
LESSOPEN=|lesspipe.sh %s
INFOPATH=/usr/share/info
LADSPA_PATH=/usr/lib/ladspa
SANE_CONFIG_DIR=/etc/sane.d
_=/usr/bin/env
Do you see any environment variable that may affect behaviour of libvirtd?
Could you provide a GDB backtrace of the segfault? The syslog entry only
says that it crashed in libc, that's not enough information to
debug the segfault.
Unfortunately, I can't find a related core file in the system. In fact, core
file is not generated. I'll also try to fix this out and come back to the
list.
Thanks a lot,
Emre
9:02 a.m.
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
On Thu, Jun 10, 2010 at 2:05 PM, Matthias Bolte
<matthias.bolte(a)googlemail.com> wrote:
>
> 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> > Dear list,
> >
> > I'm trying to package libvirt 0.8.1 for our distribution, Pardus 2009.2.
> > libvirt is installed perfectly normal, and libvirtd runs OK when I start
> > it
> > in a console using root account.
> >
> > However, when I start libvirtd as a service, with the same parameters,
> > through the normal service startup functions, it segfaults.
> >
> > The services in Pardus 2009.2 are started using a management backend
> > which
> > works with python and service start/stop scripts are python based.
> >
> > For libvirt, it's the following:
> > http://svn.pardus.org.tr/pardus/playground/ozan/libvirt/comar/service.py
> >
> > Whatever I did, I couldn't find why libvirt is crashing. It works normal
> > when I run it from console with exactly the same parameters. Here's an
> > earlier syslog section ending with the crash:
> >
>
> There are some things to consider:
>
> - Did you use the exact same commandline as the initscript when
> testing manually?
Yes. In fact, the only parameter passed is the --daemon parameter with
current configuration.
With absolute path as the initscript?
/usr/sbin/libvirtd --daemon --config /etc/libvirt/libvirtd.conf
Assuming LIBVIRTD_ARGS is empty in the initscript.
>
> - Did you make sure to use the same environment variable configuration
> when starting libvirtd manually, compared to the initscript?
Here's the environment of the root user, I will try to find out the
environment of the service script:
MANPATH=/usr/local/share/man:/usr/share/man:/opt/sun-jre/man:/usr/kde/4/share/man
HOSTNAME=EMRE
SHELL=/bin/bash
TERM=linux
XDG_SESSION_COOKIE=3d6ade2bb28141896f3212d64bf41670-1276174999.886063-1263776093
HUSHLOGIN=FALSE
LC_ALL=en_US.UTF-8
USER=root
LS_COLORS= ...
GUILE_LOAD_PATH=/usr/share/guile/1.8
MC_ENV=/usr/share/mc/bin/mc.sh
PAGER=/usr/bin/less
CONFIG_PROTECT_MASK=/etc/texmf/web2c /etc/texmf/language.dat.d
/etc/texmf/language.def.d /etc/texmf/updmap.d
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/opt/sun-jre/bin:/usr/kde/4/sbin:/usr/kde/4/bin
I asked about the environment variables and the commandline because
you have /usr/local/sbin befreo /usr/bin in PATH. So you might have
two libvirtds installed, one in /usr/local/sbin and one in /usr/sbin.
The initscript explicitly starts the one in /usr/sbin. If you just
start libvirtd manually without an absolute path then you'll start the
one in /usr/local/sbin. This might explain why you cannot reproduce
the segfault manually, but it doesn't explain why the segfault
happens.
> Could you provide a GDB backtrace of the segfault? The syslog
entry only
> says that it crashed in libc, that's not enough information to
> debug the segfault.
Unfortunately, I can't find a related core file in the system. In fact, core
file is not generated. I'll also try to fix this out and come back to the
list.
Getting a backtrace would be simpler if you could reproduce the
problem manually. In that case you could just start libvirtd in GDB.
But getting a backtrace from a coredump will work too.
Matthias
12:57 p.m.
On Thu, Jun 10, 2010 at 5:02 PM, Matthias Bolte <
matthias.bolte(a)googlemail.com> wrote:
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> On Thu, Jun 10, 2010 at 2:05 PM, Matthias Bolte
> <matthias.bolte(a)googlemail.com> wrote:
>>
>> 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
>> > Dear list,
>> >
>> > I'm trying to package libvirt 0.8.1 for our distribution, Pardus
2009.2.
>> > libvirt is installed perfectly normal, and libvirtd runs OK when I
start
>> > it
>> > in a console using root account.
>> >
>> > However, when I start libvirtd as a service, with the same parameters,
>> > through the normal service startup functions, it segfaults.
>> >
>> > The services in Pardus 2009.2 are started using a management backend
>> > which
>> > works with python and service start/stop scripts are python based.
>> >
>> > For libvirt, it's the following:
>> >
http://svn.pardus.org.tr/pardus/playground/ozan/libvirt/comar/service.py
>> >
>> > Whatever I did, I couldn't find why libvirt is crashing. It works
normal
>> > when I run it from console with exactly the same parameters. Here's an
>> > earlier syslog section ending with the crash:
>> >
>>
>> There are some things to consider:
>>
>> - Did you use the exact same commandline as the initscript when
>> testing manually?
>
> Yes. In fact, the only parameter passed is the --daemon parameter with
> current configuration.
With absolute path as the initscript?
/usr/sbin/libvirtd --daemon --config /etc/libvirt/libvirtd.conf
Assuming LIBVIRTD_ARGS is empty in the initscript.
Yes, if you check the script service.py, you'll see. We start libvirtd with
the absolute path and exactly the above parameters. The conf file itself is
the default one.
>>
>> - Did you make sure to use the same environment variable configuration
>> when starting libvirtd manually, compared to the initscript?
>
> Here's the environment of the root user, I will try to find out the
> environment of the service script:
>
>
>
MANPATH=/usr/local/share/man:/usr/share/man:/opt/sun-jre/man:/usr/kde/4/share/man
> HOSTNAME=EMRE
> SHELL=/bin/bash
> TERM=linux
>
XDG_SESSION_COOKIE=3d6ade2bb28141896f3212d64bf41670-1276174999.886063-1263776093
> HUSHLOGIN=FALSE
> LC_ALL=en_US.UTF-8
> USER=root
> LS_COLORS= ...
> GUILE_LOAD_PATH=/usr/share/guile/1.8
> MC_ENV=/usr/share/mc/bin/mc.sh
> PAGER=/usr/bin/less
> CONFIG_PROTECT_MASK=/etc/texmf/web2c /etc/texmf/language.dat.d
> /etc/texmf/language.def.d /etc/texmf/updmap.d
>
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/opt/sun-jre/bin:/usr/kde/4/sbin:/usr/kde/4/bin
I asked about the environment variables and the commandline because
you have /usr/local/sbin befreo /usr/bin in PATH. So you might have
two libvirtds installed, one in /usr/local/sbin and one in /usr/sbin.
The initscript explicitly starts the one in /usr/sbin. If you just
start libvirtd manually without an absolute path then you'll start the
one in /usr/local/sbin. This might explain why you cannot reproduce
the segfault manually, but it doesn't explain why the segfault
happens.
There's no other installation of libvirt in the system. I can also reproduce
the same thing in all Pardus machines, so I believe it's something in
libvirt not doing well with something else in our service init mechanisms.
>> Could you provide a GDB backtrace of the segfault? The syslog entry only
>> says that it crashed in libc, that's not enough information to
>> debug the segfault.
>
> Unfortunately, I can't find a related core file in the system. In fact,
core
> file is not generated. I'll also try to fix this out and come back to the
> list.
>
Getting a backtrace would be simpler if you could reproduce the
problem manually. In that case you could just start libvirtd in GDB.
But getting a backtrace from a coredump will work too.
I can't reproduce the segfault when I run it manually. It only happens when
it's run from this python script. I will try to initialize gdb inside the
script and connect remotely to the gdb session, but it's getting a bit over
my debugging capabilities :) For example, I don't know how to assign the
symbols and source code etc from the package build directory to gdb.
Thanks a lot for your support Matthias!
Emre
1:07 p.m.
On Thu, Jun 10, 2010 at 08:57:15PM +0300, Emre Erenoglu wrote:
On Thu, Jun 10, 2010 at 5:02 PM, Matthias Bolte <
matthias.bolte(a)googlemail.com> wrote:
> 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> The initscript explicitly starts the one in /usr/sbin. If you just
> start libvirtd manually without an absolute path then you'll start the
> one in /usr/local/sbin. This might explain why you cannot reproduce
> the segfault manually, but it doesn't explain why the segfault
> happens.
>
There's no other installation of libvirt in the system. I can also reproduce
the same thing in all Pardus machines, so I believe it's something in
libvirt not doing well with something else in our service init mechanisms.
I guess I'd put money on some environment variable causing trouble.
It could be a *missing* environment variable that we expect to always
be set, or something like that
> >> Could you provide a GDB backtrace of the segfault? The
syslog entry only
> >> says that it crashed in libc, that's not enough information to
> >> debug the segfault.
> >
> > Unfortunately, I can't find a related core file in the system. In fact,
> core
> > file is not generated. I'll also try to fix this out and come back to the
> > list.
> >
>
> Getting a backtrace would be simpler if you could reproduce the
> problem manually. In that case you could just start libvirtd in GDB.
> But getting a backtrace from a coredump will work too.
>
I can't reproduce the segfault when I run it manually. It only happens when
it's run from this python script. I will try to initialize gdb inside the
script and connect remotely to the gdb session, but it's getting a bit over
my debugging capabilities :) For example, I don't know how to assign the
symbols and source code etc from the package build directory to gdb.
Try creating a wrapper script, eg
mv /usr/sbin/libvirtd /usr/sbin/libvirtd.real
cat > /usr/sbin/libvirtd <<EOF
#!/bin/sh
cd /tmp
ulimited -c unlimited
exec /usr/sbin/libvirtd.real
EOF
chmod +x /usr/sbin/libvirtd
That will hopefully give you a core dump in /tmp you can get get a
stack trace from
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
2:19 p.m.
On Thu, Jun 10, 2010 at 9:07 PM, Daniel P. Berrange <berrange(a)redhat.com>wrote:
On Thu, Jun 10, 2010 at 08:57:15PM +0300, Emre Erenoglu wrote:
> On Thu, Jun 10, 2010 at 5:02 PM, Matthias Bolte <
> matthias.bolte(a)googlemail.com> wrote:
>
> > 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> > The initscript explicitly starts the one in /usr/sbin. If you just
> > start libvirtd manually without an absolute path then you'll start the
> > one in /usr/local/sbin. This might explain why you cannot reproduce
> > the segfault manually, but it doesn't explain why the segfault
> > happens.
> >
>
> There's no other installation of libvirt in the system. I can also
reproduce
> the same thing in all Pardus machines, so I believe it's something in
> libvirt not doing well with something else in our service init
mechanisms.
I guess I'd put money on some environment variable causing trouble.
It could be a *missing* environment variable that we expect to always
be set, or something like that
Hi Daniel, thanks for your message. Yes, I did a small script file as you
suggested and found out this environment while libvirtd was run:
DBUS_STARTER_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=6c515f612162b05d554b59cd4c112d43
KRB5_KTNAME=/etc/libvirt/krb5.tab
PWD=/
DBUS_STARTER_BUS_TYPE=system
SHLVL=1
_=/usr/bin/env
This looks very weak compared to the standard root environment that I pasted
in my earlier message.
> > >> Could you provide a GDB backtrace of the segfault?
The syslog entry
only
> > >> says that it crashed in libc, that's not enough information to
> > >> debug the segfault.
> > >
> > > Unfortunately, I can't find a related core file in the system. In
fact,
> > core
> > > file is not generated. I'll also try to fix this out and come back to
the
> > > list.
> > >
> >
> > Getting a backtrace would be simpler if you could reproduce the
> > problem manually. In that case you could just start libvirtd in GDB.
> > But getting a backtrace from a coredump will work too.
> >
> I can't reproduce the segfault when I run it manually. It only happens
when
> it's run from this python script. I will try to initialize gdb inside the
> script and connect remotely to the gdb session, but it's getting a bit
over
> my debugging capabilities :) For example, I don't know how to assign the
> symbols and source code etc from the package build directory to gdb.
Try creating a wrapper script, eg
mv /usr/sbin/libvirtd /usr/sbin/libvirtd.real
cat > /usr/sbin/libvirtd <<EOF
#!/bin/sh
cd /tmp
ulimited -c unlimited
exec /usr/sbin/libvirtd.real
EOF
chmod +x /usr/sbin/libvirtd
That will hopefully give you a core dump in /tmp you can get get a
stack trace from
Yes, I got the core file with the script. However, when I open the core file
with gdb, and use bt command to get the backtrace, the only thing it tells
me is this:
Core was generated by `/usr/sbin/libvirtd --daemon'.
Program terminated with signal 11, Segmentation fault.
#0 0xb73ed8f3 in ?? ()
(gdb) bt
Cannot access memory at address 0x810b9db
Maybe I don't know enough of debugging as I know I have to see the code
lines (somehow) at this segfault point. Could you guide me on that?
Thanks,
Br,
Emre
2:35 p.m.
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
On Thu, Jun 10, 2010 at 9:07 PM, Daniel P. Berrange
<berrange(a)redhat.com>
wrote:
>
> On Thu, Jun 10, 2010 at 08:57:15PM +0300, Emre Erenoglu wrote:
> > On Thu, Jun 10, 2010 at 5:02 PM, Matthias Bolte <
> > matthias.bolte(a)googlemail.com> wrote:
> >
> > > 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> > > The initscript explicitly starts the one in /usr/sbin. If you just
> > > start libvirtd manually without an absolute path then you'll start the
> > > one in /usr/local/sbin. This might explain why you cannot reproduce
> > > the segfault manually, but it doesn't explain why the segfault
> > > happens.
> > >
> >
> > There's no other installation of libvirt in the system. I can also
> > reproduce
> > the same thing in all Pardus machines, so I believe it's something in
> > libvirt not doing well with something else in our service init
> > mechanisms.
>
> I guess I'd put money on some environment variable causing trouble.
> It could be a *missing* environment variable that we expect to always
> be set, or something like that
Hi Daniel, thanks for your message. Yes, I did a small script file as you
suggested and found out this environment while libvirtd was run:
DBUS_STARTER_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=6c515f612162b05d554b59cd4c112d43
KRB5_KTNAME=/etc/libvirt/krb5.tab
PWD=/
DBUS_STARTER_BUS_TYPE=system
SHLVL=1
_=/usr/bin/env
This looks very weak compared to the standard root environment that I pasted
in my earlier message.
No PATH? I bet there is code in libvirt that assumes getenv("PATH")
will be != NULL.
Could you try to add PATH to the environment. It can be empty, doesn't
matter. Just make sure it's there, so getenv("PATH") returns an empty
string instead of NULL.
>
> > > >> Could you provide a GDB backtrace of the segfault? The syslog
entry
> > > >> only
> > > >> says that it crashed in libc, that's not enough information
to
> > > >> debug the segfault.
> > > >
> > > > Unfortunately, I can't find a related core file in the system. In
> > > > fact,
> > > core
> > > > file is not generated. I'll also try to fix this out and come
back
> > > > to the
> > > > list.
> > > >
> > >
> > > Getting a backtrace would be simpler if you could reproduce the
> > > problem manually. In that case you could just start libvirtd in GDB.
> > > But getting a backtrace from a coredump will work too.
> > >
> > I can't reproduce the segfault when I run it manually. It only happens
> > when
> > it's run from this python script. I will try to initialize gdb inside
> > the
> > script and connect remotely to the gdb session, but it's getting a bit
> > over
> > my debugging capabilities :) For example, I don't know how to assign
> > the
> > symbols and source code etc from the package build directory to gdb.
>
> Try creating a wrapper script, eg
>
> mv /usr/sbin/libvirtd /usr/sbin/libvirtd.real
> cat > /usr/sbin/libvirtd <<EOF
> #!/bin/sh
> cd /tmp
> ulimited -c unlimited
> exec /usr/sbin/libvirtd.real
> EOF
> chmod +x /usr/sbin/libvirtd
>
> That will hopefully give you a core dump in /tmp you can get get a
> stack trace from
Yes, I got the core file with the script. However, when I open the core file
with gdb, and use bt command to get the backtrace, the only thing it tells
me is this:
Core was generated by `/usr/sbin/libvirtd --daemon'.
Program terminated with signal 11, Segmentation fault.
#0 0xb73ed8f3 in ?? ()
(gdb) bt
Cannot access memory at address 0x810b9db
Maybe I don't know enough of debugging as I know I have to see the code
lines (somehow) at this segfault point. Could you guide me on that?
Thanks,
Br,
Emre
Strange backtrace. Maybe there is heap corruption going on so that GDB
can't make sense out of it anymore.
I'll do some research about the PATH usage in libvirt now.
Matthias
2:38 p.m.
On Thu, Jun 10, 2010 at 10:35 PM, Matthias Bolte <
matthias.bolte(a)googlemail.com> wrote:
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> On Thu, Jun 10, 2010 at 9:07 PM, Daniel P. Berrange <berrange(a)redhat.com
>
> wrote:
>>
>> On Thu, Jun 10, 2010 at 08:57:15PM +0300, Emre Erenoglu wrote:
>> > On Thu, Jun 10, 2010 at 5:02 PM, Matthias Bolte <
>> > matthias.bolte(a)googlemail.com> wrote:
>> >
>> > > 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
>> > > The initscript explicitly starts the one in /usr/sbin. If you just
>> > > start libvirtd manually without an absolute path then you'll
start
the
>> > > one in /usr/local/sbin. This might explain why you cannot reproduce
>> > > the segfault manually, but it doesn't explain why the segfault
>> > > happens.
>> > >
>> >
>> > There's no other installation of libvirt in the system. I can also
>> > reproduce
>> > the same thing in all Pardus machines, so I believe it's something in
>> > libvirt not doing well with something else in our service init
>> > mechanisms.
>>
>> I guess I'd put money on some environment variable causing trouble.
>> It could be a *missing* environment variable that we expect to always
>> be set, or something like that
>
> Hi Daniel, thanks for your message. Yes, I did a small script file as you
> suggested and found out this environment while libvirtd was run:
>
>
DBUS_STARTER_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=6c515f612162b05d554b59cd4c112d43
> KRB5_KTNAME=/etc/libvirt/krb5.tab
> PWD=/
> DBUS_STARTER_BUS_TYPE=system
> SHLVL=1
> _=/usr/bin/env
>
> This looks very weak compared to the standard root environment that I
pasted
> in my earlier message.
No PATH? I bet there is code in libvirt that assumes getenv("PATH")
will be != NULL.
Could you try to add PATH to the environment. It can be empty, doesn't
matter. Just make sure it's there, so getenv("PATH") returns an empty
string instead of NULL.
I just did exactly what you said by the same instinct, ie added the PATH
environment variable, and, nailed it down! It works! wow!
>
>> > > >> Could you provide a GDB backtrace of the segfault? The
syslog
entry
>> > > >> only
>> > > >> says that it crashed in libc, that's not enough
information to
>> > > >> debug the segfault.
>> > > >
>> > > > Unfortunately, I can't find a related core file in the
system. In
>> > > > fact,
>> > > core
>> > > > file is not generated. I'll also try to fix this out and come
back
>> > > > to the
>> > > > list.
>> > > >
>> > >
>> > > Getting a backtrace would be simpler if you could reproduce the
>> > > problem manually. In that case you could just start libvirtd in GDB.
>> > > But getting a backtrace from a coredump will work too.
>> > >
>> > I can't reproduce the segfault when I run it manually. It only happens
>> > when
>> > it's run from this python script. I will try to initialize gdb inside
>> > the
>> > script and connect remotely to the gdb session, but it's getting a bit
>> > over
>> > my debugging capabilities :) For example, I don't know how to assign
>> > the
>> > symbols and source code etc from the package build directory to gdb.
>>
>> Try creating a wrapper script, eg
>>
>> mv /usr/sbin/libvirtd /usr/sbin/libvirtd.real
>> cat > /usr/sbin/libvirtd <<EOF
>> #!/bin/sh
>> cd /tmp
>> ulimited -c unlimited
>> exec /usr/sbin/libvirtd.real
>> EOF
>> chmod +x /usr/sbin/libvirtd
>>
>> That will hopefully give you a core dump in /tmp you can get get a
>> stack trace from
>
> Yes, I got the core file with the script. However, when I open the core
file
> with gdb, and use bt command to get the backtrace, the only thing it
tells
> me is this:
>
> Core was generated by `/usr/sbin/libvirtd --daemon'.
> Program terminated with signal 11, Segmentation fault.
> #0 0xb73ed8f3 in ?? ()
> (gdb) bt
> Cannot access memory at address 0x810b9db
>
> Maybe I don't know enough of debugging as I know I have to see the code
> lines (somehow) at this segfault point. Could you guide me on that?
>
> Thanks,
>
> Br,
> Emre
>
Strange backtrace. Maybe there is heap corruption going on so that GDB
can't make sense out of it anymore.
I'll do some research about the PATH usage in libvirt now.
OK. I guess it's used to find the dhcp daemon, iptables etc. Other service
scripts seem to work happily without this PATH, but I'll ask developers to
add it to the python service environment to make sure it works fine.
Thanks again Matthias, Daniel! I'm a happy guy now :)
Emre Erenoglu
2:48 p.m.
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
On Thu, Jun 10, 2010 at 10:35 PM, Matthias Bolte
<matthias.bolte(a)googlemail.com> wrote:
>
> 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> > On Thu, Jun 10, 2010 at 9:07 PM, Daniel P. Berrange
> > <berrange(a)redhat.com>
> > wrote:
> >>
> >> On Thu, Jun 10, 2010 at 08:57:15PM +0300, Emre Erenoglu wrote:
> >> > On Thu, Jun 10, 2010 at 5:02 PM, Matthias Bolte <
> >> > matthias.bolte(a)googlemail.com> wrote:
> >> >
> >> > > 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> >> > > The initscript explicitly starts the one in /usr/sbin. If you
just
> >> > > start libvirtd manually without an absolute path then you'll
start
> >> > > the
> >> > > one in /usr/local/sbin. This might explain why you cannot
reproduce
> >> > > the segfault manually, but it doesn't explain why the
segfault
> >> > > happens.
> >> > >
> >> >
> >> > There's no other installation of libvirt in the system. I can also
> >> > reproduce
> >> > the same thing in all Pardus machines, so I believe it's something
in
> >> > libvirt not doing well with something else in our service init
> >> > mechanisms.
> >>
> >> I guess I'd put money on some environment variable causing trouble.
> >> It could be a *missing* environment variable that we expect to always
> >> be set, or something like that
> >
> > Hi Daniel, thanks for your message. Yes, I did a small script file as
> > you
> > suggested and found out this environment while libvirtd was run:
> >
> >
> >
DBUS_STARTER_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=6c515f612162b05d554b59cd4c112d43
> > KRB5_KTNAME=/etc/libvirt/krb5.tab
> > PWD=/
> > DBUS_STARTER_BUS_TYPE=system
> > SHLVL=1
> > _=/usr/bin/env
> >
> > This looks very weak compared to the standard root environment that I
> > pasted
> > in my earlier message.
>
> No PATH? I bet there is code in libvirt that assumes getenv("PATH")
> will be != NULL.
>
> Could you try to add PATH to the environment. It can be empty, doesn't
> matter. Just make sure it's there, so getenv("PATH") returns an empty
> string instead of NULL.
I just did exactly what you said by the same instinct, ie added the PATH
environment variable, and, nailed it down! It works! wow!
That confirms our assumption.
> >>
> >> > > >> Could you provide a GDB backtrace of the segfault? The
syslog
> >> > > >> entry
> >> > > >> only
> >> > > >> says that it crashed in libc, that's not enough
information to
> >> > > >> debug the segfault.
> >> > > >
> >> > > > Unfortunately, I can't find a related core file in the
system. In
> >> > > > fact,
> >> > > core
> >> > > > file is not generated. I'll also try to fix this out and
come
> >> > > > back
> >> > > > to the
> >> > > > list.
> >> > > >
> >> > >
> >> > > Getting a backtrace would be simpler if you could reproduce the
> >> > > problem manually. In that case you could just start libvirtd in
> >> > > GDB.
> >> > > But getting a backtrace from a coredump will work too.
> >> > >
> >> > I can't reproduce the segfault when I run it manually. It only
> >> > happens
> >> > when
> >> > it's run from this python script. I will try to initialize gdb
inside
> >> > the
> >> > script and connect remotely to the gdb session, but it's getting a
> >> > bit
> >> > over
> >> > my debugging capabilities :) For example, I don't know how to
assign
> >> > the
> >> > symbols and source code etc from the package build directory to gdb.
> >>
> >> Try creating a wrapper script, eg
> >>
> >> mv /usr/sbin/libvirtd /usr/sbin/libvirtd.real
> >> cat > /usr/sbin/libvirtd <<EOF
> >> #!/bin/sh
> >> cd /tmp
> >> ulimited -c unlimited
> >> exec /usr/sbin/libvirtd.real
> >> EOF
> >> chmod +x /usr/sbin/libvirtd
> >>
> >> That will hopefully give you a core dump in /tmp you can get get a
> >> stack trace from
> >
> > Yes, I got the core file with the script. However, when I open the core
> > file
> > with gdb, and use bt command to get the backtrace, the only thing it
> > tells
> > me is this:
> >
> > Core was generated by `/usr/sbin/libvirtd --daemon'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0xb73ed8f3 in ?? ()
> > (gdb) bt
> > Cannot access memory at address 0x810b9db
> >
> > Maybe I don't know enough of debugging as I know I have to see the code
> > lines (somehow) at this segfault point. Could you guide me on that?
> >
> > Thanks,
> >
> > Br,
> > Emre
> >
>
> Strange backtrace. Maybe there is heap corruption going on so that GDB
> can't make sense out of it anymore.
>
> I'll do some research about the PATH usage in libvirt now.
OK. I guess it's used to find the dhcp daemon, iptables etc. Other service
scripts seem to work happily without this PATH, but I'll ask developers to
add it to the python service environment to make sure it works fine.
Thanks again Matthias, Daniel! I'm a happy guy now :)
Emre Erenoglu
Yes, libvirt tries to discover that binaries via the PATH.
The utility function virFindFileInPath used the result of
getenv("PATH") without checking it for NULL. I'll post a patch for
that in a bit.
Matthias
3:01 p.m.
2010/6/10 Matthias Bolte <matthias.bolte(a)googlemail.com>:
2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
> On Thu, Jun 10, 2010 at 10:35 PM, Matthias Bolte
...
>
> OK. I guess it's used to find the dhcp daemon, iptables etc. Other service
> scripts seem to work happily without this PATH, but I'll ask developers to
> add it to the python service environment to make sure it works fine.
>
> Thanks again Matthias, Daniel! I'm a happy guy now :)
>
> Emre Erenoglu
>
Yes, libvirt tries to discover that binaries via the PATH.
The utility function virFindFileInPath used the result of
getenv("PATH") without checking it for NULL. I'll post a patch for
that in a bit.
Matthias
As an additional note: you'll need to have PATH set even when this bug
is fixed, because libvirtd discovers the QEMU binaries and the other
relevant binaries via the PATH. So you should define PATH for libvirtd
in a way that it can find those binaries.
Matthias
4:45 p.m.
On Thu, Jun 10, 2010 at 11:01 PM, Matthias Bolte <
matthias.bolte(a)googlemail.com> wrote:
2010/6/10 Matthias Bolte <matthias.bolte(a)googlemail.com>:
> 2010/6/10 Emre Erenoglu <erenoglu(a)gmail.com>:
>> On Thu, Jun 10, 2010 at 10:35 PM, Matthias Bolte
...
>>
>> OK. I guess it's used to find the dhcp daemon, iptables etc. Other
service
>> scripts seem to work happily without this PATH, but I'll ask developers
to
>> add it to the python service environment to make sure it works fine.
>>
>> Thanks again Matthias, Daniel! I'm a happy guy now :)
>>
>> Emre Erenoglu
>>
>
> Yes, libvirt tries to discover that binaries via the PATH.
>
> The utility function virFindFileInPath used the result of
> getenv("PATH") without checking it for NULL. I'll post a patch for
> that in a bit.
>
> Matthias
>
As an additional note: you'll need to have PATH set even when this bug
is fixed, because libvirtd discovers the QEMU binaries and the other
relevant binaries via the PATH. So you should define PATH for libvirtd
in a way that it can find those binaries.
OK, I'll make sure the PATH is there. I'm fixing the service script now.
--
Emre
5280
days inactive
5280
days old
11 comments
3 participants
participants (3)
-
Daniel P. Berrange -
Emre Erenoglu -
Matthias Bolte