<meta http-equiv="content-type" content="text/html; charset=utf-8"><span class="Apple-style-span" style="border-collapse: collapse; color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: 13px; ">This is the log. I called "crm node fence node4" and this restarted node4, then shut down node3, then lost quorum and information about all running resources (crm status printed no resources configures while there are 6 groups configured).<div>
<br></div><div>-------------------------<br><div><br></div><div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=node4, magic=NA, cib=0.633.101) : Transient attribute: update</div>
<div>Mar 04 12:36:40 node1 crmd: [2717]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: do_state_transition: All 4 cluster nodes are eligible to run resources.</div>
<div>Mar 04 12:36:40 node1 crmd: [2717]: info: do_pe_invoke: Query 1938: Requesting the current CIB: S_POLICY_ENGINE</div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: do_pe_invoke_callback: Invoking the PE: query=1938, ref=pe_calc-dc-1299260200-1188, seq=400, quorate=1</div>
<div>Mar 04 12:36:40 node1 pengine: [2716]: notice: unpack_config: On loss of CCM Quorum: Ignore</div><div>Mar 04 12:36:40 node1 pengine: [2716]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0</div>
<div>Mar 04 12:36:40 node1 pengine: [2716]: WARN: pe_fence_node: Node node4 will be fenced because termination was requested</div><div>Mar 04 12:36:40 node1 pengine: [2716]: WARN: determine_online_status: Node node4 is unclean</div>
<div>Mar 04 12:36:40 node1 pengine: [2716]: info: determine_online_status: Node node2 is online</div><div>....</div><div>Mar 04 12:36:40 node1 pengine: [2716]: WARN: stage6: Scheduling Node node4 for <span class="il" style="background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(255, 189, 93); color: rgb(34, 34, 34); background-position: initial initial; background-repeat: initial initial; ">STONITH</span></div>
<div>Mar 04 12:36:40 node1 pengine: [2716]: info: native_stop_constraints: st-node3_stop_0 is implicit after node4 is fenced</div><div>....</div><div>Mar 04 12:36:40 node1 pengine: [2716]: notice: LogActions: Leave resource st-node4 (Started node3)</div>
<div>Mar 04 12:36:40 node1 pengine: [2716]: notice: LogActions: Move resource st-node3 (Started node4 -> node2)</div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]</div>
<div>Mar 04 12:36:40 node1 crmd: [2717]: info: unpack_graph: Unpacked transition 471: 6 actions in 6 synapses</div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: do_te_invoke: Processing graph 471 (ref=pe_calc-dc-1299260200-1188) derived from /var/lib/pengine/pe-warn-7.bz2</div>
<div>Mar 04 12:36:40 node1 crmd: [2717]: info: te_pseudo_action: Pseudo action 87 fired and confirmed</div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: te_rsc_command: Initiating action 88: start st-node3_start_0 on node2</div>
<div>Mar 04 12:36:40 node1 pengine: [2716]: WARN: process_pe_message: Transition 471: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-7.bz2</div><div>Mar 04 12:36:40 node1 pengine: [2716]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.</div>
<div>Mar 04 12:36:40 node1 crmd: [2717]: info: match_graph_event: Action st-node3_start_0 (88) confirmed on node2 (rc=0)</div><div>Mar 04 12:36:40 node1 crmd: [2717]: info: te_pseudo_action: Pseudo action 89 fired and confirmed</div>
<div>Mar 04 12:36:40 node1 crmd: [2717]: info: te_fence_node: Executing reboot fencing operation (91) on node4 (timeout=60000)</div><div>Mar 04 12:36:40 node1 stonithd: [2712]: info: client tengine [pid: 2717] requests a <span class="il" style="background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(255, 189, 93); color: rgb(34, 34, 34); background-position: initial initial; background-repeat: initial initial; ">STONITH</span>operation RESET on node node4</div>
<div>...</div><div>Mar 04 12:36:40 node1 stonithd: [2712]: info: we can't manage node4, broadcast request to other nodes</div><div>Mar 04 12:36:40 node1 stonithd: [2712]: info: Broadcasting the message succeeded: require others to <span class="il" style="background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(255, 189, 93); color: rgb(34, 34, 34); background-position: initial initial; background-repeat: initial initial; ">stonith</span> node node4.</div>
<div>Mar 04 12:36:43 node1 cib: [2713]: info: ais_dispatch: Membership 404: quorum retained</div><div>Mar 04 12:36:43 node1 cib: [2713]: info: crm_update_peer: Node node4: id=67152064 state=lost (new) addr=r(0) ip(192.168.0.4) votes=1 born=388 seen=400 proc=00000000000000000000000000013312</div>
<div>Mar 04 12:36:43 node1 crmd: [2717]: info: ais_dispatch: Membership 404: quorum retained</div><div>Mar 04 12:36:43 node1 crmd: [2717]: info: ais_status_callback: status: node4 is now lost (was member)</div><div>Mar 04 12:36:43 node1 crmd: [2717]: info: crm_update_peer: Node node4: id=67152064 state=lost (new) addr=r(0) ip(192.168.0.4) votes=1 born=388 seen=400 proc=00000000000000000000000000013312</div>
<div>Mar 04 12:36:43 node1 crmd: [2717]: info: erase_node_from_join: Removed node node4 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1</div><div>Mar 04 12:36:43 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1939, version=0.633.102): ok (rc=0)</div>
<div>Mar 04 12:36:43 node1 crmd: [2717]: info: crm_ais_dispatch: Setting expected votes to 4</div><div>Mar 04 12:36:43 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/1942, version=0.633.103): ok (rc=0)</div>
<div>Mar 04 12:36:46 node1 stonithd: [2712]: info: Succeeded to <span class="il" style="background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(255, 189, 93); color: rgb(34, 34, 34); background-position: initial initial; background-repeat: initial initial; ">STONITH</span> the node node4: optype=RESET. whodoit: node3</div>
<div>Mar 04 12:36:46 node1 stonithd: [2712]: info: Node node3 fenced node node4: result=SUCCEEDED.</div><div>...</div><div>Mar 04 12:36:46 node1 pengine: [2716]: info: process_pe_message: Transition 472: PEngine Input stored in: /var/lib/pengine/pe-input-3427.bz2</div>
<div>Mar 04 12:37:25 node1 crmd: [2717]: info: process_graph_event: Detected action sites-httpd_monitor_30000 from a different transition: 466 vs. 472</div><div>Mar 04 12:37:25 node1 crmd: [2717]: info: abort_transition_graph: process_graph_event:462 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=sites-httpd_monitor_30000, magic=2:-2;50:466:0:56d43a96-f91d-4456-840c-8030f2ffd6ac, cib=0.633.107) : Old event</div>
<div>Mar 04 12:37:25 node1 crmd: [2717]: WARN: update_failcount: Updating failcount for sites-httpd on node2 after failed monitor: rc=-2 (update=value++, time=1299260245)</div><div>Mar 04 12:37:25 node1 crmd: [2717]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]</div>
<div>...</div><div>Mar 04 12:37:25 node1 crmd: [2717]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]</div><div>Mar 04 12:37:25 node1 crmd: [2717]: info: unpack_graph: Unpacked transition 473: 11 actions in 11 synapses</div>
<div>Mar 04 12:37:25 node1 crmd: [2717]: info: do_te_invoke: Processing graph 473 (ref=pe_calc-dc-1299260245-1192) derived from /var/lib/pengine/pe-input-3428.bz2</div><div>Mar 04 12:37:25 node1 crmd: [2717]: info: te_pseudo_action: Pseudo action 31 fired and confirmed</div>
<div>Mar 04 12:37:25 node1 crmd: [2717]: info: te_rsc_command: Initiating action 27: stop sites-nginx_stop_0 on node2</div><div>Mar 04 12:37:26 node1 pengine: [2716]: info: process_pe_message: Transition 473: PEngine Input stored in: /var/lib/pengine/pe-input-3428.bz2</div>
<div>Mar 04 12:37:26 node1 crmd: [2717]: info: match_graph_event: Action sites-nginx_stop_0 (27) confirmed on node2 (rc=0)</div><div>Mar 04 12:37:26 node1 crmd: [2717]: info: te_rsc_command: Initiating action 4: stop sites-httpd_stop_0 on node2</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: info: handle_shutdown_request: Creating shutdown request for node3 (state=S_TRANSITION_ENGINE)</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: cib_process_shutdown_req: Shutdown REQ from node3</div>
<div>Mar 04 12:37:31 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_shutdown_req for section 'all' (origin=node3/node3/(null), version=0.633.110): ok (rc=0)</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: ais_dispatch: Membership 404: quorum retained</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: info: ais_dispatch: Membership 404: quorum retained</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: crm_update_peer: Node node3: id=50374848 state=member addr=r(0) ip(192.168.0.3) votes=1 born=400 seen=404 proc=00000000000000000000000000000002 (new)</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: info: crm_update_peer: Node node3: id=50374848 state=member addr=r(0) ip(192.168.0.3) votes=1 born=400 seen=404 proc=00000000000000000000000000000002 (new)</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1952, version=0.633.111): ok (rc=0)</div>
<div>Mar 04 12:37:31 node1 cib: [2713]: notice: ais_dispatch: Membership 408: quorum lost</div><div>Mar 04 12:37:31 node1 crmd: [2717]: info: crm_ais_dispatch: Setting expected votes to 4</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: crm_update_peer: Node node3: id=50374848 state=lost (new) addr=r(0) ip(192.168.0.3) votes=1 born=400 seen=404 proc=00000000000000000000000000000002</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: notice: ais_dispatch: Membership 408: quorum lost</div><div>Mar 04 12:37:31 node1 crmd: [2717]: info: ais_status_callback: status: node3 is now lost (was member)</div><div>Mar 04 12:37:31 node1 crmd: [2717]: info: crm_update_peer: Node node3: id=50374848 state=lost (new) addr=r(0) ip(192.168.0.3) votes=1 born=400 seen=404 proc=00000000000000000000000000000002</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: info: erase_node_from_join: Removed node node3 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/1955, version=0.633.112): ok (rc=0)</div>
<div>Mar 04 12:37:31 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1956, version=0.633.112): ok (rc=0)</div><div>Mar 04 12:37:31 node1 cib: [2713]: info: log_data_element: cib:diff: - <cib have-quorum="1" admin_epoch="0" epoch="633" num_updates="113" /></div>
<div>Mar 04 12:37:31 node1 cib: [2713]: info: log_data_element: cib:diff: + <cib have-quorum="0" admin_epoch="0" epoch="634" num_updates="1" /></div><div>Mar 04 12:37:31 node1 cib: [2713]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/1958, version=0.634.1): ok (rc=0)</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: info: crm_ais_dispatch: Setting expected votes to 4</div><div>Mar 04 12:37:31 node1 crmd: [2717]: WARN: match_down_event: No match for shutdown action on node3</div><div>Mar 04 12:37:31 node1 crmd: [2717]: info: te_update_diff: <span class="il" style="background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: rgb(255, 189, 93); color: rgb(34, 34, 34); background-position: initial initial; background-repeat: initial initial; ">Stonith</span>/shutdown of node3 not matched</div>
<div>Mar 04 12:37:31 node1 crmd: [2717]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=0, tag=node_state, id=node3, magic=NA, cib=0.633.113) : Node failure</div></div></div><div><br>
</div></span>-- <br>Regards, Pentarh Udi<br>