[Pacemaker] Segmentation fault stopping Heartbeat

[Dominik Klein]

Andrew Beekhof wrote:
> You don't have gdb installed on the machine?
> I really need the stack trace but for some reason hb_report cant get it.

I think I'm facing the same problem and my hb_report cannot get the 
coredumps either. It also says:
/usr/share/heartbeat/utillib.sh: line 300: 29428 Terminated 
  crmadmin -D > /dev/null 2>&1
while running.

I have the coredumps on my machine though and will try to run them 
through gdb and post them:

This is the corefile from these log messages, but I have a whole bunch 
of more corefiles ...

Oct 30 13:09:04 dktest2 heartbeat: [14152]: WARN: Managed 
/usr/lib/heartbeat/crmd process 14506 killed by signal 11 [SIGSEGV - 
Segmentation violation].
Oct 30 13:09:04 dktest2 heartbeat: [14152]: ERROR: Managed 
/usr/lib/heartbeat/crmd process 14506 dumped core
Oct 30 13:09:04 dktest2 heartbeat: [14152]: ERROR: Client 
/usr/lib/heartbeat/crmd (pid=14506) killed by signal 11.
Oct 30 13:09:04 dktest2 heartbeat: [14152]: ERROR: Client 
/usr/lib/heartbeat/crmd "respawning too fast"

gdb /usr/lib/heartbeat/crmd /var/lib/heartbeat/cores/hacluster/core.14506
GNU gdb 6.6.50.20070726-cvs 

Copyright (C) 2007 Free Software Foundation, Inc. 

GDB is free software, covered by the GNU General Public License, and you 
are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions. 

There is absolutely no warranty for GDB.  Type "show warranty" for 
details.
This GDB was configured as "i586-suse-linux"... 

Using host libthread_db library "/lib/libthread_db.so.1". 


warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libcrmcommon.so.2...done.
Loaded symbols for /usr/lib/libcrmcommon.so.2
Reading symbols from /usr/lib/libpe_rules.so.2...done.
Loaded symbols for /usr/lib/libpe_rules.so.2
Reading symbols from /usr/lib/libcib.so.1...done.
Loaded symbols for /usr/lib/libcib.so.1
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libstonithd.so.0...done.
Loaded symbols for /usr/lib/libstonithd.so.0
Reading symbols from /usr/lib/liblrm.so.2...done.
Loaded symbols for /usr/lib/liblrm.so.2
Reading symbols from /usr/lib/libtransitioner.so.1...done.
Loaded symbols for /usr/lib/libtransitioner.so.1
Reading symbols from /usr/lib/libcrmcluster.so.1...done.
Loaded symbols for /usr/lib/libcrmcluster.so.1
Reading symbols from /usr/lib/libhbclient.so.1...done.
Loaded symbols for /usr/lib/libhbclient.so.1
Reading symbols from /usr/lib/libccmclient.so.1...done.
Loaded symbols for /usr/lib/libccmclient.so.1
Reading symbols from /usr/lib/libgnutls.so.13...done.
Loaded symbols for /usr/lib/libgnutls.so.13
Reading symbols from /lib/libgcrypt.so.11...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /lib/libgpg-error.so.0...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /usr/lib/libplumb.so.2...done.
Loaded symbols for /usr/lib/libplumb.so.2
Reading symbols from /usr/lib/libpils.so.2...done.
Loaded symbols for /usr/lib/libpils.so.2
Reading symbols from /lib/libbz2.so.1...done.
Loaded symbols for /lib/libbz2.so.1
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /usr/lib/libxslt.so.1...done.
Loaded symbols for /usr/lib/libxslt.so.1
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libglib-2.0.so.0...done.
Loaded symbols for /usr/lib/libglib-2.0.so.0
Reading symbols from /usr/lib/libpcre.so.0...done.
Loaded symbols for /usr/lib/libpcre.so.0
Reading symbols from /usr/lib/libltdl.so.3...done.
Loaded symbols for /usr/lib/libltdl.so.3
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libaudit.so.0...done.
Loaded symbols for /lib/libaudit.so.0
Core was generated by `/usr/lib/heartbeat/crmd'.
Program terminated with signal 11, Segmentation fault.
#0  0x08070391 in tengine_stonith_connection_destroy (user_data=0x0) at 
te_utils.c:49
49          transition_graph->transition_timeout = active_timeout;
(gdb) where
#0  0x08070391 in tengine_stonith_connection_destroy (user_data=0x0) at 
te_utils.c:49
#1  0xb7caa089 in G_CH_destroy_int (source=0x8097a90) at GSource.c:663
#2  0xb790d04d in ?? () from /usr/lib/libglib-2.0.so.0
#3  0x08097a90 in ?? ()
#4  0x08049050 in ?? ()
#5  0x00000000 in ?? ()

Does this help?

Regards
Dominik