I solve this problem!<br><br><br>On one node in log i found follow error message.<br><br>slv009 .... peer is not p art of our cluster<br><br>So i stop pacemaker in that host (i use v1 for pacemaker):<br><br>/etc/pacemaker stop<br>
/etc/corosync stop <br><br><br>Then remove all cib info from /var/lib/heatbeat/crm and cleanup /var/lib/pengine dir. thean restart clsuer on that node. And vuala all begin working as expected.<br><br><br>But i still have question why this happens??? Why nodes begin think that other nodes are not the part of cluster???<br>
<br><br><div class="gmail_quote">2012/2/24 ruslan usifov <span dir="ltr"><<a href="mailto:ruslan.usifov@gmail.com">ruslan.usifov@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello<br><br>I have 3 nodes cluster setup. After upgrade OS, i get that one node parmanently on OFFLINE state.<br><br><br>OS: ubuntu 10.0.4<br>pacemaker: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c<br><br><br><br>on OFFLINE node i see in log follow:<br>
<br>Feb 24 20:27:45 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>Feb 24 20:27:45 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:28:05 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>
Feb 24 20:28:05 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>Feb 24 20:28:05 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>
timer_popped ]<br>Feb 24 20:28:05 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>tion_count_vote ]<br>Feb 24 20:28:05 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>
Feb 24 20:28:05 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:28:25 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>Feb 24 20:28:25 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>
Feb 24 20:28:25 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>timer_popped ]<br>Feb 24 20:28:25 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>
tion_count_vote ]<br>Feb 24 20:28:25 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>Feb 24 20:28:25 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:28:45 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>
Feb 24 20:28:45 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>Feb 24 20:28:45 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>
timer_popped ]<br>Feb 24 20:28:45 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>tion_count_vote ]<br>Feb 24 20:28:45 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>
Feb 24 20:28:45 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:29:05 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>Feb 24 20:29:05 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>
Feb 24 20:29:05 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>timer_popped ]<br>Feb 24 20:29:05 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>
tion_count_vote ]<br>Feb 24 20:29:05 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>Feb 24 20:29:05 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:29:25 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>
Feb 24 20:29:25 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>Feb 24 20:29:25 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>
timer_popped ]<br>Feb 24 20:29:25 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>tion_count_vote ]<br>Feb 24 20:29:25 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>
Feb 24 20:29:25 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:29:45 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>Feb 24 20:29:45 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>
Feb 24 20:29:45 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>timer_popped ]<br>Feb 24 20:29:45 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>
tion_count_vote ]<br>Feb 24 20:29:45 slv009 crmd: [9125]: info: do_dc_release: DC role released<br>Feb 24 20:29:45 slv009 crmd: [9125]: info: do_te_control: Transitioner is now inactive<br>Feb 24 20:30:05 slv009 crmd: [9125]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)<br>
Feb 24 20:30:05 slv009 crmd: [9125]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING<br>Feb 24 20:30:05 slv009 crmd: [9125]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_<br>
timer_popped ]<br>Feb 24 20:30:05 slv009 crmd: [9125]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_elec<br>tion_count_vote ]<br><br><br>I have follow crm conf:<br>
<br>node slv008<br>node slv009<br>node slv010<br>primitive http_173.192.214.78_eth1 ocf:heartbeat:IPaddr2 \<br> params ip="173.192.214.78" nic="eth1:1" cidr_netmask="30" \<br> op monitor interval="10s"<br>
primitive http_nginx ocf:heartbeat:nginx \<br> op monitor interval="10s" timeout="120s"<br>group http http_173.192.214.78_eth1 http_nginx \<br> meta target-role="Started" is-managed="true"<br>
property $id="cib-bootstrap-options" \<br> dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \<br> cluster-infrastructure="openais" \<br> expected-quorum-votes="3" \<br>
stonith-enabled="false"<br>rsc_defaults $id="rsc-options" \<br> resource-stickiness="100"<br><br><br><br><br><br>Also i cant restart pacemaker on that node cleanly ie throw init.d script (it just hung and all) <br>
<br><br><br>
</blockquote></div><br>