Has anyone seen this or would expect to ever see this? I have a four-node-vm cluster, bring two online (by resuming the VM from paused state), but crm_mon reporting different state from each node. One node's crm_mon shows both nodes online with mysql m/s IP assigned as expected. The other node's crm_mon shows only itself online with both mysql m/s IP assigned to itself. Any ideas? Here's the crm_mon output:<br>
<br><br>root@vsaas-local-db01:/home/mikema# crm_mon<br>============<br>Last updated: Mon Dec 19 05:39:51 2011<br>Stack: Heartbeat<br>Current DC: vsaas-local-db02 (f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) - partition WITHOUT quorum<br>
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd<br>4 Nodes configured, unknown expected votes<br>2 Resources configured.<br>============<br><br>Online: [ vsaas-local-db02 vsaas-local-db01 ]<br>OFFLINE: [ vsaas-local-ws02 vsaas-local-ws01 ]<br>
<br>sql-master-ip (ocf::heartbeat:IPaddr): Started vsaas-local-db01<br>sql-slave-ip (ocf::heartbeat:IPaddr): Started vsaas-local-db02<br><br><br><br>root@vsaas-local-db02:/home/mikema# crm_mon<br>============<br>
Last updated: Mon Dec 19 05:35:41 2011<br>Stack: Heartbeat<br>Current DC: vsaas-local-db01 (be9daa82-1366-4ff6-b059-e90972d51682) - partition WITHOUT quorum<br>Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd<br>4 Nodes configured, unknown expected votes<br>
2 Resources configured.<br>============<br><br>Online: [ vsaas-local-db01 ]<br>OFFLINE: [ vsaas-local-db02 vsaas-local-ws02 vsaas-local-ws01 ]<br><br>sql-master-ip (ocf::heartbeat:IPaddr): Started vsaas-local-db01<br>
sql-slave-ip (ocf::heartbeat:IPaddr): Started vsaas-local-db01<br><br><br>Trying to restart heartbeat on vsaas-local-db02, but it just hangs:<br><br>root@vsaas-local-db01:/home/mikema# /etc/init.d/heartbeat restart<br>
Stopping High-Availability services: <br><br><br>Here's the log from the time I resumed play on the VM to trying to restart heartbeat:<br><br>Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.383108] ADDRCONF(NETDEV_UP): eth0: link is not ready<br>
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.400989] usb 2-2.1: USB disconnect, address 4<br>Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.433745] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None<br>
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.435708] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query 54: Requesting the current CIB: S_POLICY_ENGINE<br>
Dec 19 05:33:03 vsaas-local-db01 cib: [3183]: info: cib_stats: Processed 44 operations (3181.00us average, 0% utilization) in the last 10min<br>Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: determine_online_status: Node vsaas-local-db01 is online<br>
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: determine_online_status: Node vsaas-local-ws01 is online<br>Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: native_print: sql-master-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01<br>
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: native_print: sql-slave-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01<br>Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: colocation_match: sql-master-ip: Rolling back scores from sql-slave-ip (1, -100)<br>
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave resource sql-master-ip#011(Started vsaas-local-db01)<br>Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave resource sql-slave-ip#011(Started vsaas-local-db01)<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke_callback: Invoking the PE: query=54, ref=pe_calc-dc-1324301583-34, seq=7, quorate=0<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: unpack_graph: Unpacked transition 1: 0 actions in 0 synapses<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1324301583-34) derived from /var/lib/pengine/pe-input-1.bz2<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: run_graph: ====================================================<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: notice: run_graph: Transition 1 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1.bz2): Complete<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: te_graph_trigger: Transition 1 is now complete<br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: notify_crmd: Transition 1 status: done - <null><br>Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition: Starting PEngine Recheck Timer<br>Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: process_pe_message: Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-1.bz2<br>
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.653755] usb 2-2.1: new full speed USB device using uhci_hcd and address 5<br>Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.770096] usb 2-2.1: configuration #1 chosen from 1 choice<br>
Dec 19 05:33:06 vsaas-local-db01 heartbeat: [3167]: WARN: node vsaas-local-ws01: is dead<br>Dec 19 05:33:06 vsaas-local-db01 heartbeat: [3167]: info: Link vsaas-local-ws01:eth0 dead.<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: notice: crmd_ha_status_callback: Status update: Node vsaas-local-ws01 now has status [dead] (DC=true)<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crm_update_peer_proc: vsaas-local-ws01.ais is now offline<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: WARN: match_down_event: No match for shutdown action on d969970d-1439-4a47-bf60-3b8d3fc4558b<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: te_update_diff: Stonith/shutdown of d969970d-1439-4a47-bf60-3b8d3fc4558b not matched<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=d969970d-1439-4a47-bf60-3b8d3fc4558b, magic=NA, cib=0.15.3) : Node failure<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no mbr_track info<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: instance=8, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=5<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback: Quorum lost after event=INVALID (id=8)<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: INVALID: trans=8, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=5<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: #011CURRENT: vsaas-local-db01 [nodeid=0, born=8]<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: #011LOST: vsaas-local-ws01 [nodeid=2, born=6]<br>Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ais_status_callback: status: vsaas-local-ws01 is now lost (was member)<br>Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no mbr_track info<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crm_update_peer: Node vsaas-local-ws01: id=2 state=lost (new) addr=(null) votes=-1 born=6 seen=7 proc=00000000000000000000000000000200<br>
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: erase_node_from_join: Removed node vsaas-local-ws01 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1<br>
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: instance=8, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=5<br>Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha: Requesting the list of configured nodes<br>
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback: Processing CCM event=INVALID (id=8)<br>Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: crm_update_peer: Node vsaas-local-ws01: id=2 state=lost (new) addr=(null) votes=-1 born=6 seen=7 proc=00000000000000000000000000000302<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query 58: Requesting the current CIB: S_POLICY_ENGINE<br>Dec 19 05:33:08 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/56, version=0.15.3): ok (rc=0)<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke_callback: Invoking the PE: query=58, ref=pe_calc-dc-1324301588-36, seq=8, quorate=0<br>Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: determine_online_status: Node vsaas-local-db01 is online<br>
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: native_print: sql-master-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01<br>Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: native_print: sql-slave-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01<br>
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: colocation_match: sql-master-ip: Rolling back scores from sql-slave-ip (1, -100)<br>Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave resource sql-master-ip#011(Started vsaas-local-db01)<br>
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave resource sql-slave-ip#011(Started vsaas-local-db01)<br>Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: unpack_graph: Unpacked transition 2: 0 actions in 0 synapses<br>Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1324301588-36) derived from /var/lib/pengine/pe-input-2.bz2<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: run_graph: ====================================================<br>Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: notice: run_graph: Transition 2 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-2.bz2): Complete<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: te_graph_trigger: Transition 2 is now complete<br>Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: notify_crmd: Transition 2 status: done - <null><br>Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition: Starting PEngine Recheck Timer<br>Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: process_pe_message: Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-2.bz2<br>
Dec 19 05:34:11 vsaas-local-db01 heartbeat: [3167]: info: Link vsaas-local-db02:eth0 up.<br>Dec 19 05:34:11 vsaas-local-db01 cib: [3183]: WARN: cib_peer_callback: Discarding cib_apply_diff message (8b59) from vsaas-local-db02: not in our membership<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode =vsaas-local-db02, fromnode's ackseq = 0<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686, lowseq=35643,ackseq=35643,lastmsg=185<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: For information on cluster partitions, See URL: <a href="http://linux-ha.org/wiki/Split_Brain">http://linux-ha.org/wiki/Split_Brain</a><br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: WARN: Deadtime value may be too small.<br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: See FAQ for information on tuning deadtime.<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: URL: <a href="http://linux-ha.org/wiki/FAQ#Heavy_Load">http://linux-ha.org/wiki/FAQ#Heavy_Load</a><br>
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: WARN: Late heartbeat: Node vsaas-local-db02: interval 122790 ms<br>Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: Status update for node vsaas-local-db02: status active<br>
Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: notice: crmd_ha_status_callback: Status update: Node vsaas-local-db02 now has status [active] (DC=true)<br>Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: info: crm_update_peer_proc: vsaas-local-db02.ais is now online<br>
Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: WARN: crmd_ha_msg_callback: Ignoring HA message (op=vote) from vsaas-local-db02: not in our membership list (size=1)<br>Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no mbr_track info<br>Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: instance=9, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=5<br>
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback: Quorum lost after event=INVALID (id=9)<br>Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: INVALID: trans=9, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=5<br>
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: #011CURRENT: vsaas-local-db01 [nodeid=0, born=9]<br>Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha: Requesting the list of configured nodes<br>
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no mbr_track info<br>Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: instance=9, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=5<br>Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback: Processing CCM event=INVALID (id=9)<br>
Dec 19 05:34:15 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/60, version=0.15.5): ok (rc=0)<br>Dec 19 05:34:15 vsaas-local-db01 cib: [3183]: WARN: cib_peer_callback: Discarding cib_apply_diff message (8b62) from vsaas-local-db02: not in our membership<br>
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no mbr_track info<br>Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: instance=10, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=6<br>Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback: Quorum lost after event=INVALID (id=10)<br>
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: INVALID: trans=10, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=6<br>Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: #011CURRENT: vsaas-local-db01 [nodeid=0, born=1]<br>
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: #011CURRENT: vsaas-local-db02 [nodeid=1, born=10]<br>Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail: #011NEW: vsaas-local-db02 [nodeid=1, born=10]<br>
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ais_status_callback: status: vsaas-local-db02 is now member (was lost)<br>Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: crm_update_peer: Node vsaas-local-db02: id=1 state=member (new) addr=(null) votes=-1 born=10 seen=10 proc=00000000000000000000000000000202<br>
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha: Requesting the list of configured nodes<br>Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no mbr_track info<br>Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm<br>Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: instance=10, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=6<br>
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback: Processing CCM event=INVALID (id=10)<br>Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: crm_update_peer: Node vsaas-local-db02: id=1 state=member (new) addr=(null) votes=-1 born=10 seen=10 proc=00000000000000000000000000000302<br>
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsaas-local-db02']/lrm (origin=local/crmd/62, version=0.15.6): ok (rc=0)<br>
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='vsaas-local-db02']/transient_attributes (origin=local/crmd/63, version=0.15.7): ok (rc=0)<br>
Dec 19 05:34:17 vsaas-local-db01 cib: [3183]: WARN: cib_process_diff: Diff 0.14.20 -> 0.15.1 not applied to 0.15.7: current "epoch" is greater than required<br>Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=sql-master-ip_monitor_0, magic=0:7;4:3:7:e132d41b-0185-4355-9cbc-e0acb43058c5, cib=0.15.6) : Resource op removal<br>
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vsaas-local-db02']/lrm": ok (rc=0)<br>Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1, magic=NA, cib=0.15.7) : Transient attribute: removal<br>
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: erase_xpath_callback: Deletion of "//node_state[@uname='vsaas-local-db02']/transient_attributes": ok (rc=0)<br>Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition: Membership changed: 7 -> 10 - join restart<br>Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query 66: Requesting the current CIB: S_POLICY_ENGINE<br>
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]<br>Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC vsaas-local-db01<br>
Dec 19 05:34:17 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/64, version=0.15.7): ok (rc=0)<br>Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: join_make_offer: Making join offers based on membership 10<br>
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-3: Waiting on 2 outstanding join acks<br>Dec 19 05:34:18 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to vsaas-local-db01 (3.0.1)<br>
Dec 19 05:34:19 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>Dec 19 05:34:19 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC vsaas-local-db01<br>
Dec 19 05:34:19 vsaas-local-db01 cib: [3183]: WARN: cib_process_diff: Diff 0.15.1 -> 0.15.2 not applied to 0.15.8: current "num_updates" is greater than required<br>Dec 19 05:36:15 vsaas-local-db01 crmd: [3187]: info: crmd_ha_msg_filter: Another DC detected: vsaas-local-db02 (op=join_offer)<br>
Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>
Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/68, version=0.15.8): ok (rc=0)<br>
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/69, version=0.15.8): ok (rc=0)<br>
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/70, version=0.15.8): ok (rc=0)<br>Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/72, version=0.15.8): ok (rc=0)<br>
Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks<br>Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/74, version=0.15.8): ok (rc=0)<br>
Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to vsaas-local-db01 (3.0.1)<br>
Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 5 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ]<br>
Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC vsaas-local-db01<br>Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>
Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/77, version=0.15.8): ok (rc=0)<br>
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/78, version=0.15.8): ok (rc=0)<br>
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/79, version=0.15.8): ok (rc=0)<br>Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/81, version=0.15.8): ok (rc=0)<br>
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks<br>Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/83, version=0.15.8): ok (rc=0)<br>
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION<br>Dec 19 05:38:27 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 6 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>
Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>
Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/85, version=0.15.8): ok (rc=0)<br>
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/86, version=0.15.8): ok (rc=0)<br>
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/87, version=0.15.8): ok (rc=0)<br>Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/89, version=0.15.8): ok (rc=0)<br>
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-6: Waiting on 2 outstanding join acks<br>Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/91, version=0.15.8): ok (rc=0)<br>
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION<br>Dec 19 05:40:30 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 7 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>
Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>
Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/93, version=0.15.8): ok (rc=0)<br>
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/94, version=0.15.8): ok (rc=0)<br>
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/95, version=0.15.8): ok (rc=0)<br>Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/97, version=0.15.8): ok (rc=0)<br>
Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-7: Waiting on 2 outstanding join acks<br>Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/99, version=0.15.8): ok (rc=0)<br>
Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to vsaas-local-db01 (3.0.1)<br>
Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC vsaas-local-db01<br>
Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 8 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 9 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>
Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input I_JOIN_REQUEST from route_message() received in state S_ELECTION<br>Dec 19 05:43:03 vsaas-local-db01 cib: [3183]: info: cib_stats: Processed 54 operations (2962.00us average, 0% utilization) in the last 10min<br>
Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>
Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/102, version=0.15.8): ok (rc=0)<br>
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/103, version=0.15.8): ok (rc=0)<br>
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/104, version=0.15.8): ok (rc=0)<br>Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/106, version=0.15.8): ok (rc=0)<br>
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-8: Waiting on 2 outstanding join acks<br>Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/108, version=0.15.8): ok (rc=0)<br>
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION<br>Dec 19 05:44:38 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 10 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>
Dec 19 05:46:28 vsaas-local-db01 heartbeat: [3167]: info: killing /usr/lib/heartbeat/crmd process group 3187 with signal 15<br>Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated<br>
Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: crm_shutdown: Requesting shutdown<br>Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: do_shutdown_req: Sending shutdown request to DC: <null><br>Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: handle_shutdown_request: Creating shutdown request for vsaas-local-db01 (state=S_ELECTION)<br>
Dec 19 05:46:28 vsaas-local-db01 attrd: [3186]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (1324302388)<br>Dec 19 05:46:28 vsaas-local-db01 attrd: [3186]: info: attrd_perform_update: Sent update 97: shutdown=1324302388<br>
Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>
Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/111, version=0.15.10): ok (rc=0)<br>
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/112, version=0.15.10): ok (rc=0)<br>
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/113, version=0.15.10): ok (rc=0)<br>Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/115, version=0.15.10): ok (rc=0)<br>
Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-9: Waiting on 2 outstanding join acks<br>Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/117, version=0.15.10): ok (rc=0)<br>
Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to vsaas-local-db01 (3.0.1)<br>
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC vsaas-local-db01<br>
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 11 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 12 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input I_JOIN_REQUEST from route_message() received in state S_ELECTION<br>Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]<br>
Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: info: start_subsystem: Starting sub-system "pengine"<br>Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem: Client pengine already running as pid 3557<br>
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking over DC status for this partition<br>Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/O mode<br>Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/120, version=0.15.10): ok (rc=0)<br>
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite: We are now in R/W mode<br>Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/121, version=0.15.10): ok (rc=0)<br>
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/122, version=0.15.10): ok (rc=0)<br>Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/124, version=0.15.10): ok (rc=0)<br>
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all: join-10: Waiting on 2 outstanding join acks<br>Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/126, version=0.15.10): ok (rc=0)<br>
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: config_query_callback: Checking for expired actions every 900000ms<br>Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]<br>
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION<br>Dec 19 05:48:44 vsaas-local-db01 crmd: [3187]: info: do_election_count_vote: Election 13 (owner: f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)<br>
root@vsaas-local-db01:/home/mikema# <br><br><br>