[Pacemaker] ccm returning with exit code 100 and system rebooting

[akshay punja]

Please let me know if any one has solved this issue. CCM exiting with return
code 100 and system rebooting

On Mon, Jan 17, 2011 at 1:29 PM, akshay punja <akshay.punja at gmail.com>wrote:

> Hi All,
>
> We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with
> heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment.
>
> Node : we are using two node in a cluster and hosting a bunch of
> application on the HA.
>
> We are seeing a strange rebooting of one of the nodes *Managed
> /usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could
> be possible issue and how could we fix it.
> *
> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes
> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: false
> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is
> disabled --enabling logging daemon is recommended
> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info:
> **************************
> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration validated.
> Starting heartbeat 3.0.2
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version 3.0.2
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation:
> 1293182645
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write
> socket priority set to IPTOS_LOWDELAY on eth0
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send
> socket to device: eth0
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound
> receive socket to device: eth0
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on
> port 694 interface eth0 to 172.21.52.135
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
> G_main_add_TriggerHandler: Added signal manual handler
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
> G_main_add_TriggerHandler: Added signal manual handler
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
> G_main_add_SignalHandler: Added signal handler for signal 17
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler
> parameters.: Operation not permitted
> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set to:
> 'up'
> Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set scheduler
> parameters.: Operation not permitted
> Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set scheduler
> parameters.: Operation not permitted
> Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set scheduler
> parameters.: Operation not permitted
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating
> status to active
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set to:
> 'active'
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> "/usr/lib/heartbeat/ccm" (100,101)
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> "/usr/lib/heartbeat/cib" (100,101)
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> "/usr/lib/heartbeat/lrmd -r" (0,0)
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> "/usr/lib/heartbeat/stonithd" (0,0)
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> "/usr/lib/heartbeat/attrd" (100,101)
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> "/usr/lib/heartbeat/crmd" (100,101)
> Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting
> "/usr/lib/heartbeat/ccm" as uid 100  gid 101 (pid 19576)
> Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting
> "/usr/lib/heartbeat/cib" as uid 100  gid 101 (pid 19577)
> Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting
> "/usr/lib/heartbeat/lrmd -r" as uid 0  gid 0 (pid 19578)
> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> Added signal handler for signal 15
> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> Added signal handler for signal 17
> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps
> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> Added signal handler for signal 10
> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> Added signal handler for signal 12
> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started.
> Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting
> "/usr/lib/heartbeat/stonithd" as uid 0  gid 0 (pid 19579)
> Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting
> "/usr/lib/heartbeat/attrd" as uid 100  gid 101 (pid 19580)
> *Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed
> /usr/lib/heartbeat/ccm process 19576 exited with return code 100.
> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system.
> Reason: /usr/lib/heartbeat/ccm*
> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler:
> Added signal handler for signal 10
> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler:
> Added signal handler for signal 12
> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect:
> Connecting to Heartbeat
> Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting
> "/usr/lib/heartbeat/crmd" as uid 100  gid 101 (pid 19581)
> Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS
> FAILED: /sbin/reboot -nf returned 0
> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: register_heartbeat_conn:
> Cannot sign on with heartbeat:
> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to
> cluster
> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:
> /usr/lib/heartbeat/stonithd abnormally abort.
> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown:
> Master Control process died.
> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with
> SIGTERM
> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with
> SIGTERM
> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with
> SIGTERM
> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP
> dead): Killing ourselves.*
>
> Regards,
> Akshay
>
>
> *
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110118/c92d1ecd/attachment.htm>