[Pacemaker] ccm rebooting with exit code 100

[akshay punja]

Hi All,

We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with
heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment.

Node : we are using two node in a cluster and hosting a bunch of application
on the HA.

We are seeing a strange rebooting of one of the nodes *Managed
/usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could
be possible issue and how could we fix it.
*
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: false
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is
disabled --enabling logging daemon is recommended
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info:
**************************
Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration validated.
Starting heartbeat 3.0.2
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version 3.0.2
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation:
1293182645
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write socket
priority set to IPTOS_LOWDELAY on eth0
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send
socket to device: eth0
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound
receive socket to device: eth0
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on
port 694 interface eth0 to 172.21.52.135
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
G_main_add_TriggerHandler: Added signal manual handler
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
G_main_add_TriggerHandler: Added signal manual handler
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler
parameters.: Operation not permitted
Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set to:
'up'
Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set scheduler
parameters.: Operation not permitted
Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set scheduler
parameters.: Operation not permitted
Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set scheduler
parameters.: Operation not permitted
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating
status to active
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set to:
'active'
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
"/usr/lib/heartbeat/ccm" (100,101)
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
"/usr/lib/heartbeat/cib" (100,101)
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
"/usr/lib/heartbeat/lrmd -r" (0,0)
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
"/usr/lib/heartbeat/stonithd" (0,0)
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
"/usr/lib/heartbeat/attrd" (100,101)
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
"/usr/lib/heartbeat/crmd" (100,101)
Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting
"/usr/lib/heartbeat/ccm" as uid 100  gid 101 (pid 19576)
Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting
"/usr/lib/heartbeat/cib" as uid 100  gid 101 (pid 19577)
Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting
"/usr/lib/heartbeat/lrmd -r" as uid 0  gid 0 (pid 19578)
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started.
Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting
"/usr/lib/heartbeat/stonithd" as uid 0  gid 0 (pid 19579)
Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting
"/usr/lib/heartbeat/attrd" as uid 100  gid 101 (pid 19580)
*Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed
/usr/lib/heartbeat/ccm process 19576 exited with return code 100.
Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system.
Reason: /usr/lib/heartbeat/ccm*
Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect:
Connecting to Heartbeat
Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting
"/usr/lib/heartbeat/crmd" as uid 100  gid 101 (pid 19581)
Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS
FAILED: /sbin/reboot -nf returned 0
Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: register_heartbeat_conn:
Cannot sign on with heartbeat:
Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to
cluster
Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:
/usr/lib/heartbeat/stonithd abnormally abort.
Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown:
Master Control process died.
Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with
SIGTERM
Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with
SIGTERM
Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with
SIGTERM
Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP
dead): Killing ourselves.*

Regards,
Akshay


*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110117/c85ef30c/attachment.htm>