[Pacemaker] Pacemaker core dumps
Xavier Lashmar
xlashmar at uottawa.ca
Thu May 2 13:37:59 UTC 2013
Ah, finally got it.
Core was generated by `/usr/libexec/pacemaker/crmd'.
Program terminated with signal 6, Aborted.
#0 0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install libtool-ltdl-2.2.6-15.5.el6.x86_64
(gdb) bt
#0 0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f81896ae085 in abort () at abort.c:92
#2 0x00007f818bb8a56b in crm_abort (file=0x7f818bba9d58 "xml.c", function=0x7f818bbab6b4 "string2xml", line=650,
assert_condition=0x7f818bbaa01a "String parsing error", do_core=<value optimized out>, do_fork=<value optimized out>) at utils.c:1073
#3 0x00007f818bb933af in string2xml (
input=0x1e745f8 "<lrmd_notify lrmd_origin=\"send_cmd_complete_notify\" lrmd_timeout=\"30000\" lrmd_rsc_interval=\"15000\" lrmd_rsc_start_delay=\"15000\" lrmd_exec_rc=\"0\" lrmd_exec_op_status=\"1\" lrmd_callid=\"2747\" lrmd_rsc_del"...) at xml.c:650
#4 0x00007f818b76a2fc in lrmd_ipc_dispatch (buffer=<value optimized out>, length=<value optimized out>, userdata=0x1e72910) at lrmd_client.c:310
#5 0x00007f818bba2e90 in mainloop_gio_callback (gio=<value optimized out>, condition=G_IO_IN, data=0x1e73be0) at mainloop.c:585
#6 0x00007f8188fbbf0e in g_main_dispatch (context=0x1d4f120) at gmain.c:1960
#7 IA__g_main_context_dispatch (context=0x1d4f120) at gmain.c:2513
#8 0x00007f8188fbf938 in g_main_context_iterate (context=0x1d4f120, block=1, dispatch=1, self=<value optimized out>) at gmain.c:2591
#9 0x00007f8188fbfd55 in IA__g_main_loop_run (loop=0x1e734a0) at gmain.c:2799
#10 0x00000000004052ce in crmd_init () at main.c:154
#11 0x00000000004055cc in main (argc=1, argv=0x7fffe77a4f88) at main.c:120
Xavier Lashmar
Analyste de Systèmes | Systems Analyst
Service étudiants, service de l'informatique et des communications/Student services, computing and communications services.
1 Nicholas Street (810)
Ottawa ON K1N 7B7
Tél. | Tel. 613-562-5800 (2120)
-----Original Message-----
From: Andrew Beekhof [mailto:andrew at beekhof.net]
Sent: Wednesday, May 1, 2013 7:07 PM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Pacemaker core dumps
On 01/05/2013, at 11:36 PM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:
> I'm not sure if anyone has run into this issue but I can't seem to
> find a debuginfo package for one of the libraries for CentOS 6.3 with
> Kernel 2.6.32-279.9.1el6.x86_64 : libtool-ltdl
>
> Here's what I get so far from the core dump, but I think it's incomplete:
>
> ...
> ...
> ...
> Reading symbols from /lib64/libfreebl3.so...
> warning: the debug information found in "/usr/lib/debug//lib64/libfreebl3.so.debug" does not match "/lib64/libfreebl3.so" (CRC mismatch).
>
> warning: the debug information found in "/usr/lib/debug/lib64/libfreebl3.so.debug" does not match "/lib64/libfreebl3.so" (CRC mismatch).
>
> Missing separate debuginfo for /lib64/libfreebl3.so
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/68/195872ecfb188389d29aaf01031a976fd18168.deb
> ug
> (no debugging symbols found)...done.
> Loaded symbols for /lib64/libfreebl3.so Reading symbols from
> /lib64/libnss_files-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_files-2.12.so Core was generated by
> `/usr/libexec/pacemaker/crmd'.
> Program terminated with signal 6, Aborted.
> #0 0x00007f81896ac8a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> Missing separate debuginfos, use: debuginfo-install
> libtool-ltdl-2.2.6-15.5.el6.x86_64
>
> Any info about either finding the right debuginfo files, or about the error itself would be greatly appreciated.
The libtool parts aren't so interesting.
Were there no other frames? (lines starting with # and a number)
>
> Xavier Lashmar
> Analyste de Systèmes | Systems Analyst Service étudiants, service de
> l'informatique et des communications/Student services, computing and communications services.
> 1 Nicholas Street (810)
> Ottawa ON K1N 7B7
> Tél. | Tel. 613-562-5800 (2120)
>
>
>
>
> -----Original Message-----
> From: Andrew Beekhof [mailto:andrew at beekhof.net]
> Sent: Monday, April 29, 2013 11:00 PM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Pacemaker core dumps
>
>
> On 30/04/2013, at 1:32 AM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:
>
>> Hello Andrew,
>>
>> Thanks for your help. We've upgrade to pacemaker 1.1.9 and still have the same issue.
>
> Thats a disappointing but useful data point.
>
>>
>> We are trying to get the core information but we are missing some debuginfo files which we are trying to get our hands on. I'll try to forward this information soon.
>
> Great
>
>>
>> Is there something we need to do to the CIB when we upgrade?
>
> No, anything that needs to happen will be done under the hood.
>
>>
>>
>> Xavier Lashmar
>> Analyste de Systèmes | Systems Analyst Service étudiants, service de
>> l'informatique et des communications/Student services, computing and communications services.
>> 1 Nicholas Street (810)
>> Ottawa ON K1N 7B7
>> Tél. | Tel. 613-562-5800 (2120)
>>
>>
>>
>> -----Original Message-----
>> From: Andrew Beekhof [mailto:andrew at beekhof.net]
>> Sent: Thursday, April 25, 2013 8:15 PM
>> To: The Pacemaker cluster resource manager
>> Subject: Re: [Pacemaker] Pacemaker core dumps
>>
>>
>> On 26/04/2013, at 10:06 AM, Andrew Beekhof <andrew at beekhof.net> wrote:
>>
>>>
>>> On 25/04/2013, at 11:59 PM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:
>>>
>>>> Following further investigation, we were able to determine that upgrading both nodes (in a two node cluster) from Pacemaker 1.1.7-6 to Pacemaker 1.1.8-7 (CentOS 6.3 or Centos 6.4) caused these errors to begin happening:
>>>
>>> Would you be able to try the 1.1.9 packages from http://www.clusterlabs.org/rpm-next to see if they are also affected?
>>>
>>>>
>>>> We were able to replicate the initiation of the errors by upgrading another cluster in the same manner. This other cluster is now experiencing the same core-dumping and errors as the previous cluster:
>>>>
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : invalid character in attribute value
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : attributes construct error
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity: line 1: parser error : Extra content at the end of the document
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: Parsing
>>>> failed (domain=1, level=3, code=5): Extra content at the end of the
>>>> document Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String start:
>>>> <lrmd_notify lrmd_origin="send_cmd_complete_notify
>>>> Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String start+688: 0000" CRM_meta_start_delay="15000"/></lrmd_notify>
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_abort: string2xml: Forked child 4182 to record non-fatal assert at xml.c:605 : String parsing error
>>
>> Also, it would be very useful if you could open up the core file for
>> 4182 and print the contents of the input passed to string2xml()
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
More information about the Pacemaker
mailing list