[Pacemaker] crm_mon reporting different state and hearbeat won't stop or restart

[gustavo panizzo <gfa>]

i think cluster is out of sync

run cibadmin -Q |grep epoch on each node and compare the output, on a
sane cluster should be the same

out of sync cluster, happened to me once, was a lot of pain 

On Mon, Dec 19, 2011 at 08:57:04AM -0500, Michael Marrotte wrote:
> Has anyone seen this or would expect to ever see this?  I have a
> four-node-vm cluster, bring two online (by resuming the VM from paused
> state), but crm_mon reporting different state from each node.   One node's
> crm_mon shows both nodes online with mysql m/s IP assigned as expected.
> The other node's crm_mon shows only itself online with both mysql m/s IP
> assigned to itself.  Any ideas?  Here's the crm_mon output:
> 
> 
> root at vsaas-local-db01:/home/mikema# crm_mon
> ============
> Last updated: Mon Dec 19 05:39:51 2011
> Stack: Heartbeat
> Current DC: vsaas-local-db02 (f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) -
> partition WITHOUT quorum
> Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
> 4 Nodes configured, unknown expected votes
> 2 Resources configured.
> ============
> 
> Online: [ vsaas-local-db02 vsaas-local-db01 ]
> OFFLINE: [ vsaas-local-ws02 vsaas-local-ws01 ]
> 
> sql-master-ip   (ocf::heartbeat:IPaddr):        Started vsaas-local-db01
> sql-slave-ip    (ocf::heartbeat:IPaddr):        Started vsaas-local-db02
> 
> 
> 
> root at vsaas-local-db02:/home/mikema# crm_mon
> ============
> Last updated: Mon Dec 19 05:35:41 2011
> Stack: Heartbeat
> Current DC: vsaas-local-db01 (be9daa82-1366-4ff6-b059-e90972d51682) -
> partition WITHOUT quorum
> Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
> 4 Nodes configured, unknown expected votes
> 2 Resources configured.
> ============
> 
> Online: [ vsaas-local-db01 ]
> OFFLINE: [ vsaas-local-db02 vsaas-local-ws02 vsaas-local-ws01 ]
> 
> sql-master-ip   (ocf::heartbeat:IPaddr):        Started vsaas-local-db01
> sql-slave-ip    (ocf::heartbeat:IPaddr):        Started vsaas-local-db01
> 
> 
> Trying to restart heartbeat on vsaas-local-db02, but it just hangs:
> 
> root at vsaas-local-db01:/home/mikema# /etc/init.d/heartbeat restart
> Stopping High-Availability services:
> 
> 
> Here's the log from the time I resumed play on the VM to trying to restart
> heartbeat:
> 
> Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.383108]
> ADDRCONF(NETDEV_UP): eth0: link is not ready
> Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.400989] usb 2-2.1: USB
> disconnect, address 4
> Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.433745] e1000: eth0 NIC
> Link is Up 1000 Mbps Full Duplex, Flow Control: None
> Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.435708]
> ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: crm_timer_popped:
> PEngine Recheck Timer (I_PE_CALC) just popped!
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> All 2 cluster nodes are eligible to run resources.
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query
> 54: Requesting the current CIB: S_POLICY_ENGINE
> Dec 19 05:33:03 vsaas-local-db01 cib: [3183]: info: cib_stats: Processed 44
> operations (3181.00us average, 0% utilization) in the last 10min
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: unpack_config: Node
> scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info:
> determine_online_status: Node vsaas-local-db01 is online
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info:
> determine_online_status: Node vsaas-local-ws01 is online
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: native_print:
> sql-master-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: native_print:
> sql-slave-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: colocation_match:
> sql-master-ip: Rolling back scores from sql-slave-ip (1, -100)
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
> resource sql-master-ip#011(Started vsaas-local-db01)
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
> resource sql-slave-ip#011(Started vsaas-local-db01)
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke_callback:
> Invoking the PE: query=54, ref=pe_calc-dc-1324301583-34, seq=7, quorate=0
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
> input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: unpack_graph: Unpacked
> transition 1: 0 actions in 0 synapses
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_te_invoke:
> Processing graph 1 (ref=pe_calc-dc-1324301583-34) derived from
> /var/lib/pengine/pe-input-1.bz2
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: run_graph:
> ====================================================
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: notice: run_graph:
> Transition 1 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pengine/pe-input-1.bz2): Complete
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: te_graph_trigger:
> Transition 1 is now complete
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: notify_crmd:
> Transition 1 status: done - <null>
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> Starting PEngine Recheck Timer
> Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: process_pe_message:
> Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-1.bz2
> Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.653755] usb 2-2.1: new full
> speed USB device using uhci_hcd and address 5
> Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.770096] usb 2-2.1:
> configuration #1 chosen from 1 choice
> Dec 19 05:33:06 vsaas-local-db01 heartbeat: [3167]: WARN: node
> vsaas-local-ws01: is dead
> Dec 19 05:33:06 vsaas-local-db01 heartbeat: [3167]: info: Link
> vsaas-local-ws01:eth0 dead.
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: notice:
> crmd_ha_status_callback: Status update: Node vsaas-local-ws01 now has
> status [dead] (DC=true)
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crm_update_peer_proc:
> vsaas-local-ws01.ais is now offline
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: WARN: match_down_event: No
> match for shutdown action on d969970d-1439-4a47-bf60-3b8d3fc4558b
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: te_update_diff:
> Stonith/shutdown of d969970d-1439-4a47-bf60-3b8d3fc4558b not matched
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info:
> abort_transition_graph: te_update_diff:191 - Triggered transition abort
> (complete=1, tag=node_state, id=d969970d-1439-4a47-bf60-3b8d3fc4558b,
> magic=NA, cib=0.15.3) : Node failure
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no
> mbr_track info
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event:
> instance=8, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=5
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback:
> Quorum lost after event=INVALID (id=8)
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> INVALID: trans=8, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=5
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> #011CURRENT: vsaas-local-db01 [nodeid=0, born=8]
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> #011LOST:    vsaas-local-ws01 [nodeid=2, born=6]
> Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ais_status_callback:
> status: vsaas-local-ws01 is now lost (was member)
> Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no
> mbr_track info
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crm_update_peer: Node
> vsaas-local-ws01: id=2 state=lost (new) addr=(null) votes=-1 born=6 seen=7
> proc=00000000000000000000000000000200
> Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: erase_node_from_join:
> Removed node vsaas-local-ws01 from join calculations: welcomed=0
> itegrated=0 finalized=0 confirmed=1
> Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event:
> instance=8, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=5
> Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha:
> Requesting the list of configured nodes
> Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback:
> Processing CCM event=INVALID (id=8)
> Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: crm_update_peer: Node
> vsaas-local-ws01: id=2 state=lost (new) addr=(null) votes=-1 born=6 seen=7
> proc=00000000000000000000000000000302
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> All 1 cluster nodes are eligible to run resources.
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query
> 58: Requesting the current CIB: S_POLICY_ENGINE
> Dec 19 05:33:08 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section nodes (origin=local/crmd/56,
> version=0.15.3): ok (rc=0)
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke_callback:
> Invoking the PE: query=58, ref=pe_calc-dc-1324301588-36, seq=8, quorate=0
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: unpack_config: Node
> scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info:
> determine_online_status: Node vsaas-local-db01 is online
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: native_print:
> sql-master-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: native_print:
> sql-slave-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: colocation_match:
> sql-master-ip: Rolling back scores from sql-slave-ip (1, -100)
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
> resource sql-master-ip#011(Started vsaas-local-db01)
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
> resource sql-slave-ip#011(Started vsaas-local-db01)
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
> input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: unpack_graph: Unpacked
> transition 2: 0 actions in 0 synapses
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_te_invoke:
> Processing graph 2 (ref=pe_calc-dc-1324301588-36) derived from
> /var/lib/pengine/pe-input-2.bz2
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: run_graph:
> ====================================================
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: notice: run_graph:
> Transition 2 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pengine/pe-input-2.bz2): Complete
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: te_graph_trigger:
> Transition 2 is now complete
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: notify_crmd:
> Transition 2 status: done - <null>
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> Starting PEngine Recheck Timer
> Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: process_pe_message:
> Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-2.bz2
> Dec 19 05:34:11 vsaas-local-db01 heartbeat: [3167]: info: Link
> vsaas-local-db02:eth0 up.
> Dec 19 05:34:11 vsaas-local-db01 cib: [3183]: WARN: cib_peer_callback:
> Discarding cib_apply_diff message (8b59) from vsaas-local-db02: not in our
> membership
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
> =vsaas-local-db02, fromnode's ackseq = 0
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
> lowseq=35643,ackseq=35643,lastmsg=185
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: For information
> on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: WARN: Deadtime value
> may be too small.
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: See FAQ for
> information on tuning deadtime.
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: URL:
> http://linux-ha.org/wiki/FAQ#Heavy_Load
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: WARN: Late heartbeat:
> Node vsaas-local-db02: interval 122790 ms
> Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: Status update for
> node vsaas-local-db02: status active
> Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: notice:
> crmd_ha_status_callback: Status update: Node vsaas-local-db02 now has
> status [active] (DC=true)
> Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: info: crm_update_peer_proc:
> vsaas-local-db02.ais is now online
> Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: WARN: crmd_ha_msg_callback:
> Ignoring HA message (op=vote) from vsaas-local-db02: not in our membership
> list (size=1)
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no
> mbr_track info
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event:
> instance=9, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=5
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback:
> Quorum lost after event=INVALID (id=9)
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> INVALID: trans=9, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=5
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> #011CURRENT: vsaas-local-db01 [nodeid=0, born=9]
> Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha:
> Requesting the list of configured nodes
> Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no
> mbr_track info
> Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event:
> instance=9, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=5
> Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback:
> Processing CCM event=INVALID (id=9)
> Dec 19 05:34:15 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section nodes (origin=local/crmd/60,
> version=0.15.5): ok (rc=0)
> Dec 19 05:34:15 vsaas-local-db01 cib: [3183]: WARN: cib_peer_callback:
> Discarding cib_apply_diff message (8b62) from vsaas-local-db02: not in our
> membership
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no
> mbr_track info
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event:
> instance=10, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=6
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback:
> Quorum lost after event=INVALID (id=10)
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> INVALID: trans=10, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=6
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> #011CURRENT: vsaas-local-db01 [nodeid=0, born=1]
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> #011CURRENT: vsaas-local-db02 [nodeid=1, born=10]
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
> #011NEW:     vsaas-local-db02 [nodeid=1, born=10]
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ais_status_callback:
> status: vsaas-local-db02 is now member (was lost)
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: crm_update_peer: Node
> vsaas-local-db02: id=1 state=member (new) addr=(null) votes=-1 born=10
> seen=10 proc=00000000000000000000000000000202
> Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha:
> Requesting the list of configured nodes
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no
> mbr_track info
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
> an event OC_EV_MS_INVALID from ccm
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event:
> instance=10, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=6
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback:
> Processing CCM event=INVALID (id=10)
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: crm_update_peer: Node
> vsaas-local-db02: id=1 state=member (new) addr=(null) votes=-1 born=10
> seen=10 proc=00000000000000000000000000000302
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_delete for section
> //node_state[@uname='vsaas-local-db02']/lrm (origin=local/crmd/62,
> version=0.15.6): ok (rc=0)
> Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_delete for section
> //node_state[@uname='vsaas-local-db02']/transient_attributes
> (origin=local/crmd/63, version=0.15.7): ok (rc=0)
> Dec 19 05:34:17 vsaas-local-db01 cib: [3183]: WARN: cib_process_diff: Diff
> 0.14.20 -> 0.15.1 not applied to 0.15.7: current "epoch" is greater than
> required
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info:
> abort_transition_graph: te_update_diff:267 - Triggered transition abort
> (complete=1, tag=lrm_rsc_op, id=sql-master-ip_monitor_0,
> magic=0:7;4:3:7:e132d41b-0185-4355-9cbc-e0acb43058c5, cib=0.15.6) :
> Resource op removal
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: erase_xpath_callback:
> Deletion of "//node_state[@uname='vsaas-local-db02']/lrm": ok (rc=0)
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info:
> abort_transition_graph: te_update_diff:157 - Triggered transition abort
> (complete=1, tag=transient_attributes,
> id=f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1, magic=NA, cib=0.15.7) : Transient
> attribute: removal
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: erase_xpath_callback:
> Deletion of "//node_state[@uname='vsaas-local-db02']/transient_attributes":
> ok (rc=0)
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> Membership changed: 7 -> 10 - join restart
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query
> 66: Requesting the current CIB: S_POLICY_ENGINE
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
> cause=C_FSA_INTERNAL origin=do_state_transition ]
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
> vsaas-local-db01
> Dec 19 05:34:17 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section nodes (origin=local/crmd/64,
> version=0.15.7): ok (rc=0)
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: join_make_offer:
> Making join offers based on membership 10
> Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-3: Waiting on 2 outstanding join acks
> Dec 19 05:34:18 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
> vsaas-local-db01 (3.0.1)
> Dec 19 05:34:19 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:34:19 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
> vsaas-local-db01
> Dec 19 05:34:19 vsaas-local-db01 cib: [3183]: WARN: cib_process_diff: Diff
> 0.15.1 -> 0.15.2 not applied to 0.15.8: current "num_updates" is greater
> than required
> Dec 19 05:36:15 vsaas-local-db01 crmd: [3187]: info: crmd_ha_msg_filter:
> Another DC detected: vsaas-local-db02 (op=join_offer)
> Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/68, version=0.15.8): ok (rc=0)
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/69,
> version=0.15.8): ok (rc=0)
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/70,
> version=0.15.8): ok (rc=0)
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/72, version=0.15.8): ok (rc=0)
> Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-4: Waiting on 2 outstanding join acks
> Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/74, version=0.15.8): ok (rc=0)
> Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
> vsaas-local-db01 (3.0.1)
> Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 5 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
> vsaas-local-db01
> Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/77, version=0.15.8): ok (rc=0)
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/78,
> version=0.15.8): ok (rc=0)
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/79,
> version=0.15.8): ok (rc=0)
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/81, version=0.15.8): ok (rc=0)
> Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-5: Waiting on 2 outstanding join acks
> Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/83, version=0.15.8): ok (rc=0)
> Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Dec 19 05:38:27 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 6 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/85, version=0.15.8): ok (rc=0)
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/86,
> version=0.15.8): ok (rc=0)
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/87,
> version=0.15.8): ok (rc=0)
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/89, version=0.15.8): ok (rc=0)
> Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-6: Waiting on 2 outstanding join acks
> Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/91, version=0.15.8): ok (rc=0)
> Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Dec 19 05:40:30 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 7 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/93, version=0.15.8): ok (rc=0)
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/94,
> version=0.15.8): ok (rc=0)
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/95,
> version=0.15.8): ok (rc=0)
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/97, version=0.15.8): ok (rc=0)
> Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-7: Waiting on 2 outstanding join acks
> Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/99, version=0.15.8): ok (rc=0)
> Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
> vsaas-local-db01 (3.0.1)
> Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
> vsaas-local-db01
> Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 8 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 9 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
> I_JOIN_REQUEST from route_message() received in state S_ELECTION
> Dec 19 05:43:03 vsaas-local-db01 cib: [3183]: info: cib_stats: Processed 54
> operations (2962.00us average, 0% utilization) in the last 10min
> Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/102, version=0.15.8): ok (rc=0)
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/103,
> version=0.15.8): ok (rc=0)
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/104,
> version=0.15.8): ok (rc=0)
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/106, version=0.15.8): ok (rc=0)
> Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-8: Waiting on 2 outstanding join acks
> Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/108, version=0.15.8): ok (rc=0)
> Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Dec 19 05:44:38 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 10 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:46:28 vsaas-local-db01 heartbeat: [3167]: info: killing
> /usr/lib/heartbeat/crmd process group 3187 with signal 15
> Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: crm_signal_dispatch:
> Invoking handler for signal 15: Terminated
> Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: crm_shutdown:
> Requesting shutdown
> Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: do_shutdown_req:
> Sending shutdown request to DC: <null>
> Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info:
> handle_shutdown_request: Creating shutdown request for vsaas-local-db01
> (state=S_ELECTION)
> Dec 19 05:46:28 vsaas-local-db01 attrd: [3186]: info: attrd_trigger_update:
> Sending flush op to all hosts for: shutdown (1324302388)
> Dec 19 05:46:28 vsaas-local-db01 attrd: [3186]: info: attrd_perform_update:
> Sent update 97: shutdown=1324302388
> Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/111, version=0.15.10): ok (rc=0)
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/112,
> version=0.15.10): ok (rc=0)
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/113,
> version=0.15.10): ok (rc=0)
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/115, version=0.15.10): ok (rc=0)
> Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-9: Waiting on 2 outstanding join acks
> Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/117, version=0.15.10): ok (rc=0)
> Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
> vsaas-local-db01 (3.0.1)
> Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
> vsaas-local-db01
> Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 11 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 12 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
> I_JOIN_REQUEST from route_message() received in state S_ELECTION
> Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
> Starting sub-system "pengine"
> Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
> Client pengine already running as pid 3557
> Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
> over DC status for this partition
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/O mode
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_slave_all for section 'all'
> (origin=local/crmd/120, version=0.15.10): ok (rc=0)
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
> We are now in R/W mode
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_master for section 'all' (origin=local/crmd/121,
> version=0.15.10): ok (rc=0)
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section cib (origin=local/crmd/122,
> version=0.15.10): ok (rc=0)
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/124, version=0.15.10): ok (rc=0)
> Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
> join-10: Waiting on 2 outstanding join acks
> Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
> Operation complete: op cib_modify for section crm_config
> (origin=local/crmd/126, version=0.15.10): ok (rc=0)
> Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
> Checking for expired actions every 900000ms
> Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
> State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
> cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
> Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
> I_JOIN_OFFER from route_message() received in state S_ELECTION
> Dec 19 05:48:44 vsaas-local-db01 crmd: [3187]: info:
> do_election_count_vote: Election 13 (owner:
> f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
> root at vsaas-local-db01:/home/mikema#

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


-- 
1AE0 322E B8F7 4717 BDEA  BF1D 44BB 1BA7 9F6C 6333