Hi All,<br><br>We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment.<br><br>Node : we are using two node in a cluster and hosting a bunch of application on the HA.<br>
<br>We are seeing a strange rebooting of one of the nodes <b>Managed /usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could be possible issue and how could we fix it.<br></b><br>Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes<br>
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: false<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is disabled --enabling logging daemon is recommended<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: **************************<br>
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration validated. Starting heartbeat 3.0.2<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version 3.0.2<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation: 1293182645<br>
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send socket to device: eth0<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound receive socket to device: eth0<br>
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on port 694 interface eth0 to 172.21.52.135<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: G_main_add_TriggerHandler: Added signal manual handler<br>
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: G_main_add_TriggerHandler: Added signal manual handler<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: G_main_add_SignalHandler: Added signal handler for signal 17<br>
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set to: 'up'<br>Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>
Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead<br>Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating status to active<br>Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set to: 'active'<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client "/usr/lib/heartbeat/ccm" (100,101)<br>Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client "/usr/lib/heartbeat/cib" (100,101)<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client "/usr/lib/heartbeat/lrmd -r" (0,0)<br>Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client "/usr/lib/heartbeat/stonithd" (0,0)<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client "/usr/lib/heartbeat/attrd" (100,101)<br>Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client "/usr/lib/heartbeat/crmd" (100,101)<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting "/usr/lib/heartbeat/ccm" as uid 100 gid 101 (pid 19576)<br>Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting "/usr/lib/heartbeat/cib" as uid 100 gid 101 (pid 19577)<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting "/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 19578)<br>Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: Added signal handler for signal 15<br>
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: Added signal handler for signal 17<br>Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps<br>Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: Added signal handler for signal 10<br>
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler: Added signal handler for signal 12<br>Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started.<br>Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 19579)<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting "/usr/lib/heartbeat/attrd" as uid 100 gid 101 (pid 19580)<br><b>Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed /usr/lib/heartbeat/ccm process 19576 exited with return code 100.<br>
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system. Reason: /usr/lib/heartbeat/ccm</b><br>Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler: Added signal handler for signal 10<br>
Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler: Added signal handler for signal 12<br>Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect: Connecting to Heartbeat<br>Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting "/usr/lib/heartbeat/crmd" as uid 100 gid 101 (pid 19581)<br>
Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS FAILED: /sbin/reboot -nf returned 0<br>Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: register_heartbeat_conn: Cannot sign on with heartbeat:<br>Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to cluster<br>
Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: /usr/lib/heartbeat/stonithd abnormally abort.<br>Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown: Master Control process died.<br>Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with SIGTERM<br>
Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with SIGTERM<br>Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with SIGTERM<br>Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP dead): Killing ourselves.<b><br>
<br>Regards,<br>Akshay<br><br><br></b>