[Pacemaker] Pacemaker core dumps

Thu Apr 25 20:06:45 EDT 2013

On 25/04/2013, at 11:59 PM, Xavier Lashmar <xlashmar at uottawa.ca> wrote:

> Following further investigation, we were able to determine that upgrading both nodes (in a two node cluster) from Pacemaker 1.1.7-6 to Pacemaker 1.1.8-7 (CentOS 6.3 or Centos 6.4) caused these errors to begin happening:

Would you be able to try the 1.1.9 packages from http://www.clusterlabs.org/rpm-next to see if they are also affected?

>  
> We were able to replicate the initiation of the errors by upgrading another cluster in the same manner.  This other cluster is now experiencing the same core-dumping and errors as the previous cluster:
>  
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : invalid character in attribute value
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : attributes construct error
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: Entity: line 1: parser error : Extra content at the end of the document
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error: a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is running...
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_xml_err: XML Error:                                                                                ^
> Apr 25 09:46:22 xxxx crmd[1764]:  warning: string2xml: Parsing failed (domain=1, level=3, code=5): Extra content at the end of the document
> Apr 25 09:46:22 xxxx crmd[1764]:  warning: string2xml: String start: <lrmd_notify lrmd_origin="send_cmd_complete_notify
> Apr 25 09:46:22 xxxx crmd[1764]:  warning: string2xml: String start+688: 0000" CRM_meta_start_delay="15000"/></lrmd_notify>
> Apr 25 09:46:22 xxxx crmd[1764]:    error: crm_abort: string2xml: Forked child 4182 to record non-fatal assert at xml.c:605 : String parsing error
>  
> Any assistance would be appreciated.
>  
> Xavier Lashmar
> X2120
>  
> From: Xavier Lashmar [mailto:xlashmar at uottawa.ca] 
> Sent: Tuesday, April 23, 2013 3:41 PM
> To: Pacemaker at oss.clusterlabs.org
> Subject: [Pacemaker] Pacemaker core dumps
>  
> Hello everyone,
>  
> Below is a message that my colleague tried to post to the mailing list, but somehow it didn’t make it yet, so I’m re-posting just in case it got lost somewhere:
>  
> Hi All,
>  
> I currently have a pacemaker + drbd cluster. I am getting core dumps every 15 minutes. I can’t seem to figure out what has triggered this. How do you see what is being output in the core dump? Also, in my  corosync.log I found the following errors:
>  
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: Entity: line 1: parser error : invalid character in attribute value
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error:                                                                                ^
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: Entity: line 1: parser error : attributes construct error
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error:                                                                                ^
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: Entity: line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error:                                                                                ^
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: Entity: line 1: parser error : Extra content at the end of the document
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error: e-9c6d-44bf-963b-87836613e694" lrmd_rsc_output="tomcat6 (pid 8899) is running...
> Apr 22 10:37:15 [2058]       crmd:    error: crm_xml_err:  XML Error:                                                                                ^
> Apr 22 10:37:15 [2058]       crmd:  warning: string2xml:   Parsing failed (domain=1, level=3, code=5): Extra content at the end of the document
> Apr 22 10:37:15 [2058]       crmd:  warning: string2xml:   String start: <lrmd_notify lrmd_origin="send_cmd_complete_notify
> Apr 22 10:37:15 [2058]      crmd:  warning: string2xml:   String start+645: ="30000" CRM_meta_interval="15000"/></lrmd_notify>
> Apr 22 10:37:15 [2058]      crmd:    error: crm_abort:    string2xml: Forked child 22964 to record non-fatal assert at xml.c:605 : String parsing error
>  
> So I thought the errors might be in my xml config file so I extracted the file and found no errors in there. From the log file, this string “e-9c6d-44bf-963b-87836613e694” seems to be part of the transition-key. I am not sure if that helps shed some light on the issue but any help would be appreciated.
>  
>  
> <image001.png>
> <image002.png>
> <image003.png>
> Xavier Lashmar
> Analyste de Systèmes | Systems Analyst
> Service étudiants, service de l'informatique et des communications/Student services, computing and communications services.
> 1 Nicholas Street (810)
> Ottawa ON K1N 7B7
> Tél. | Tel. 613-562-5800 (2120)
>  
>  
>  
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org