Hi,<br><br>Thanks for the help,<br><br>As suggest I have changed the crm on to respawn, after the configuration change Rebooting has stopped.<br><br>I are using tomcat, apache httpd and mysql master - slave replication, I have set this up in multiple environments and its working fine. I are see this issue only in one of the nodes and so I isolated the node to find a solution. The log file is only printing info and warn is there a way to enable debug logging too. <br>
<br><b>ERROR: Unable to set scheduler parameters.: Operation not permitted</b> - we are seeing this issue in health clusters in other environment too. I think this would not be the root cause<br> <br><br>bash-3.2# ps -ef<br>
UID PID PPID C STIME TTY TIME CMD<br>root 2662 21268 0 11:17 pts/0 00:00:00 ps -ef<br>root 1 0 0 11:01 ? 00:00:00 init [3]<br>root 20382 1 0 11:01 ? 00:00:00 /usr/sbin/syslog_adapter<br>
root 20389 1 0 11:01 ? 00:00:00 syslogd -m 0 -p /dev/log_adapted<br>root 20400 1 0 11:01 ? 00:00:00 logmgr<br>root 21236 2953 0 11:02 pts/0 00:00:00 /bin/sh /bin/console<br>root 21268 21236 0 11:02 pts/0 00:00:00 /bin/bash --login<br>
root 30746 1 0 11:12 ? 00:00:00 heartbeat: master control proces<br>root 30748 30746 0 11:12 ? 00:00:00 heartbeat: FIFO reader<br>root 30749 30746 0 11:12 ? 00:00:00 heartbeat: write: ucast eth0<br>
root 30750 30746 0 11:12 ? 00:00:00 heartbeat: read: ucast eth0<br>root 32731 30746 0 11:14 ? 00:00:00 /usr/lib/heartbeat/lrmd -r<br>root 32732 30746 0 11:14 ? 00:00:00 /usr/lib/heartbeat/stonithd<br>
<br>Log file after the configuration ifle <br>Jan 18 11:12:47 mysqlis1 heartbeat: [30745]: info: Pacemaker support: respawn<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30745]: info: Pacemaker support: false<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30745]: WARN: Logging daemon is disabled --enabling logging daemon is recommended<br>
Jan 18 11:12:47 mysqlis1 heartbeat: [30745]: info: **************************<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30745]: info: Configuration validated. Starting heartbeat 3.0.2<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: heartbeat: version 3.0.2<br>
Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: Heartbeat generation: 1293183350<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: glib: ucast: bound send socket to device: eth0<br>
Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: glib: ucast: bound receive socket to device: eth0<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: glib: ucast: started on port 694 interface eth0 to 172.21.52.135<br>
Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: G_main_add_TriggerHandler: Added signal manual handler<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: G_main_add_TriggerHandler: Added signal manual handler<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: G_main_add_SignalHandler: Added signal handler for signal 17<br>
Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>Jan 18 11:12:47 mysqlis1 heartbeat: [30746]: info: Local status now set to: 'up'<br>Jan 18 11:12:48 mysqlis1 heartbeat: [30748]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>
Jan 18 11:12:48 mysqlis1 heartbeat: [30749]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>Jan 18 11:12:48 mysqlis1 heartbeat: [30750]: ERROR: Unable to set scheduler parameters.: Operation not permitted<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: WARN: node mysql3: is dead<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Comm_now_up(): updating status to active<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Local status now set to: 'active'<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Starting child client "/usr/lib/heartbeat/ccm" (100,101)<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Starting child client "/usr/lib/heartbeat/cib" (100,101)<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Starting child client "/usr/lib/heartbeat/lrmd -r" (0,0)<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Starting child client "/usr/lib/heartbeat/stonithd" (0,0)<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Starting child client "/usr/lib/heartbeat/attrd" (100,101)<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: info: Starting child client "/usr/lib/heartbeat/crmd" (100,101)<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [32729]: info: Starting "/usr/lib/heartbeat/ccm" as uid 100 gid 101 (pid 32729)<br>Jan 18 11:14:48 mysqlis1 heartbeat: [32730]: info: Starting "/usr/lib/heartbeat/cib" as uid 100 gid 101 (pid 32730)<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [32731]: info: Starting "/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 32731)<br>Jan 18 11:14:48 mysqlis1 lrmd: [32731]: info: G_main_add_SignalHandler: Added signal handler for signal 15<br>
Jan 18 11:14:48 mysqlis1 lrmd: [32731]: info: G_main_add_SignalHandler: Added signal handler for signal 17<br>Jan 18 11:14:48 mysqlis1 lrmd: [32731]: info: enabling coredumps<br>Jan 18 11:14:48 mysqlis1 lrmd: [32731]: info: G_main_add_SignalHandler: Added signal handler for signal 10<br>
Jan 18 11:14:48 mysqlis1 lrmd: [32731]: info: G_main_add_SignalHandler: Added signal handler for signal 12<br>Jan 18 11:14:48 mysqlis1 lrmd: [32731]: info: Started.<br>Jan 18 11:14:48 mysqlis1 heartbeat: [32732]: info: Starting "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 32732)<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [32733]: info: Starting "/usr/lib/heartbeat/attrd" as uid 100 gid 101 (pid 32733)<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: WARN: Managed /usr/lib/heartbeat/ccm process 32729 exited with return code 100.<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: WARN: Managed /usr/lib/heartbeat/cib process 32730 exited with return code 100.<br>Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: WARN: Managed /usr/lib/heartbeat/attrd process 32733 exited with return code 100.<br>
Jan 18 11:14:48 mysqlis1 stonithd: [32732]: info: G_main_add_SignalHandler: Added signal handler for signal 10<br>Jan 18 11:14:48 mysqlis1 stonithd: [32732]: info: G_main_add_SignalHandler: Added signal handler for signal 12<br>
Jan 18 11:14:48 mysqlis1 stonithd: [32732]: info: crm_cluster_connect: Connecting to Heartbeat<br>Jan 18 11:14:48 mysqlis1 heartbeat: [32734]: info: Starting "/usr/lib/heartbeat/crmd" as uid 100 gid 101 (pid 32734)<br>
Jan 18 11:14:48 mysqlis1 heartbeat: [30746]: WARN: Managed /usr/lib/heartbeat/crmd process 32734 exited with return code 100.<br>Jan 18 11:14:48 mysqlis1 stonithd: [32732]: info: register_heartbeat_conn: Hostname: mysqlis1<br>
Jan 18 11:14:48 mysqlis1 stonithd: [32732]: info: register_heartbeat_conn: UUID: d26dfd2b-5412-42b5-84d2-86567676c849<br>Jan 18 11:14:48 mysqlis1 stonithd: [32732]: notice: /usr/lib/heartbeat/stonithd start up successfully.<br>
Jan 18 11:14:48 mysqlis1 stonithd: [32732]: info: G_main_add_SignalHandler: Added signal handler for signal 17<br><br>Regards,<br>Akshay<br><br><div class="gmail_quote">On Tue, Jan 18, 2011 at 2:25 PM, Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div class="im">On Tue, Jan 18, 2011 at 4:04 AM, akshay punja <<a href="mailto:akshay.punja@gmail.com">akshay.punja@gmail.com</a>> wrote:<br>
> Please let me know if any one has solved this issue.<br>
<br>
</div>Can you try "crm respawn" instead of "crm on" so the node stays up<br>
long enough to see why the ccm is unhappy.<br>
<br>
Lars, you really aught to think about changing the default behavior<br>
and adding "crm fatal" or something.<br>
<div><div></div><div class="h5"><br>
> CCM exiting with return code 100 and system rebooting<br>
><br>
> On Mon, Jan 17, 2011 at 1:29 PM, akshay punja <<a href="mailto:akshay.punja@gmail.com">akshay.punja@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Hi All,<br>
>><br>
>> We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with<br>
>> heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment.<br>
>><br>
>> Node : we are using two node in a cluster and hosting a bunch of<br>
>> application on the HA.<br>
>><br>
>> We are seeing a strange rebooting of one of the nodes Managed<br>
>> /usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could<br>
>> be possible issue and how could we fix it.<br>
>><br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support:<br>
>> false<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is<br>
>> disabled --enabling logging daemon is recommended<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info:<br>
>> **************************<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration<br>
>> validated. Starting heartbeat 3.0.2<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version<br>
>> 3.0.2<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation:<br>
>> 1293182645<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write<br>
>> socket priority set to IPTOS_LOWDELAY on eth0<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send<br>
>> socket to device: eth0<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound<br>
>> receive socket to device: eth0<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on<br>
>> port 694 interface eth0 to 172.21.52.135<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:<br>
>> G_main_add_TriggerHandler: Added signal manual handler<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:<br>
>> G_main_add_TriggerHandler: Added signal manual handler<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:<br>
>> G_main_add_SignalHandler: Added signal handler for signal 17<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set<br>
>> scheduler parameters.: Operation not permitted<br>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set<br>
>> to: 'up'<br>
>> Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set<br>
>> scheduler parameters.: Operation not permitted<br>
>> Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set<br>
>> scheduler parameters.: Operation not permitted<br>
>> Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set<br>
>> scheduler parameters.: Operation not permitted<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating<br>
>> status to active<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set<br>
>> to: 'active'<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client<br>
>> "/usr/lib/heartbeat/ccm" (100,101)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client<br>
>> "/usr/lib/heartbeat/cib" (100,101)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client<br>
>> "/usr/lib/heartbeat/lrmd -r" (0,0)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client<br>
>> "/usr/lib/heartbeat/stonithd" (0,0)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client<br>
>> "/usr/lib/heartbeat/attrd" (100,101)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client<br>
>> "/usr/lib/heartbeat/crmd" (100,101)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting<br>
>> "/usr/lib/heartbeat/ccm" as uid 100 gid 101 (pid 19576)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting<br>
>> "/usr/lib/heartbeat/cib" as uid 100 gid 101 (pid 19577)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting<br>
>> "/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 19578)<br>
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:<br>
>> Added signal handler for signal 15<br>
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:<br>
>> Added signal handler for signal 17<br>
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps<br>
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:<br>
>> Added signal handler for signal 10<br>
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:<br>
>> Added signal handler for signal 12<br>
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started.<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting<br>
>> "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 19579)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting<br>
>> "/usr/lib/heartbeat/attrd" as uid 100 gid 101 (pid 19580)<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed<br>
>> /usr/lib/heartbeat/ccm process 19576 exited with return code 100.<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system.<br>
>> Reason: /usr/lib/heartbeat/ccm<br>
>> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info:<br>
>> G_main_add_SignalHandler: Added signal handler for signal 10<br>
>> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info:<br>
>> G_main_add_SignalHandler: Added signal handler for signal 12<br>
>> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect:<br>
>> Connecting to Heartbeat<br>
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting<br>
>> "/usr/lib/heartbeat/crmd" as uid 100 gid 101 (pid 19581)<br>
>> Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS<br>
>> FAILED: /sbin/reboot -nf returned 0<br>
>> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:<br>
>> register_heartbeat_conn: Cannot sign on with heartbeat:<br>
>> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to<br>
>> cluster<br>
>> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:<br>
>> /usr/lib/heartbeat/stonithd abnormally abort.<br>
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown:<br>
>> Master Control process died.<br>
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with<br>
>> SIGTERM<br>
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with<br>
>> SIGTERM<br>
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with<br>
>> SIGTERM<br>
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP<br>
>> dead): Killing ourselves.<br>
>><br>
>> Regards,<br>
>> Akshay<br>
>><br>
>><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs:<br>
> <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
><br>
><br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
</blockquote></div><br>