[Pacemaker] crm_mon reporting different state and hearbeat won't stop or restart

[Michael Marrotte]

Has anyone seen this or would expect to ever see this?  I have a
four-node-vm cluster, bring two online (by resuming the VM from paused
state), but crm_mon reporting different state from each node.   One node's
crm_mon shows both nodes online with mysql m/s IP assigned as expected.
The other node's crm_mon shows only itself online with both mysql m/s IP
assigned to itself.  Any ideas?  Here's the crm_mon output:


root at vsaas-local-db01:/home/mikema# crm_mon
============
Last updated: Mon Dec 19 05:39:51 2011
Stack: Heartbeat
Current DC: vsaas-local-db02 (f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) -
partition WITHOUT quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
4 Nodes configured, unknown expected votes
2 Resources configured.
============

Online: [ vsaas-local-db02 vsaas-local-db01 ]
OFFLINE: [ vsaas-local-ws02 vsaas-local-ws01 ]

sql-master-ip   (ocf::heartbeat:IPaddr):        Started vsaas-local-db01
sql-slave-ip    (ocf::heartbeat:IPaddr):        Started vsaas-local-db02



root at vsaas-local-db02:/home/mikema# crm_mon
============
Last updated: Mon Dec 19 05:35:41 2011
Stack: Heartbeat
Current DC: vsaas-local-db01 (be9daa82-1366-4ff6-b059-e90972d51682) -
partition WITHOUT quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
4 Nodes configured, unknown expected votes
2 Resources configured.
============

Online: [ vsaas-local-db01 ]
OFFLINE: [ vsaas-local-db02 vsaas-local-ws02 vsaas-local-ws01 ]

sql-master-ip   (ocf::heartbeat:IPaddr):        Started vsaas-local-db01
sql-slave-ip    (ocf::heartbeat:IPaddr):        Started vsaas-local-db01


Trying to restart heartbeat on vsaas-local-db02, but it just hangs:

root at vsaas-local-db01:/home/mikema# /etc/init.d/heartbeat restart
Stopping High-Availability services:


Here's the log from the time I resumed play on the VM to trying to restart
heartbeat:

Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.383108]
ADDRCONF(NETDEV_UP): eth0: link is not ready
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.400989] usb 2-2.1: USB
disconnect, address 4
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.433745] e1000: eth0 NIC
Link is Up 1000 Mbps Full Duplex, Flow Control: None
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.435708]
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
All 2 cluster nodes are eligible to run resources.
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query
54: Requesting the current CIB: S_POLICY_ENGINE
Dec 19 05:33:03 vsaas-local-db01 cib: [3183]: info: cib_stats: Processed 44
operations (3181.00us average, 0% utilization) in the last 10min
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: unpack_config: On
loss of CCM Quorum: Ignore
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: unpack_config: Node
scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info:
determine_online_status: Node vsaas-local-db01 is online
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info:
determine_online_status: Node vsaas-local-ws01 is online
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: native_print:
sql-master-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: native_print:
sql-slave-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: colocation_match:
sql-master-ip: Rolling back scores from sql-slave-ip (1, -100)
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
resource sql-master-ip#011(Started vsaas-local-db01)
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
resource sql-slave-ip#011(Started vsaas-local-db01)
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke_callback:
Invoking the PE: query=54, ref=pe_calc-dc-1324301583-34, seq=7, quorate=0
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: unpack_graph: Unpacked
transition 1: 0 actions in 0 synapses
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_te_invoke:
Processing graph 1 (ref=pe_calc-dc-1324301583-34) derived from
/var/lib/pengine/pe-input-1.bz2
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: run_graph:
====================================================
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: notice: run_graph:
Transition 1 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-1.bz2): Complete
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: te_graph_trigger:
Transition 1 is now complete
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: notify_crmd:
Transition 1 status: done - <null>
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 19 05:33:03 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
Starting PEngine Recheck Timer
Dec 19 05:33:03 vsaas-local-db01 pengine: [3557]: info: process_pe_message:
Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-1.bz2
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.653755] usb 2-2.1: new full
speed USB device using uhci_hcd and address 5
Dec 19 05:33:03 vsaas-local-db01 kernel: [88899.770096] usb 2-2.1:
configuration #1 chosen from 1 choice
Dec 19 05:33:06 vsaas-local-db01 heartbeat: [3167]: WARN: node
vsaas-local-ws01: is dead
Dec 19 05:33:06 vsaas-local-db01 heartbeat: [3167]: info: Link
vsaas-local-ws01:eth0 dead.
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: notice:
crmd_ha_status_callback: Status update: Node vsaas-local-ws01 now has
status [dead] (DC=true)
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crm_update_peer_proc:
vsaas-local-ws01.ais is now offline
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: WARN: match_down_event: No
match for shutdown action on d969970d-1439-4a47-bf60-3b8d3fc4558b
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: te_update_diff:
Stonith/shutdown of d969970d-1439-4a47-bf60-3b8d3fc4558b not matched
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info:
abort_transition_graph: te_update_diff:191 - Triggered transition abort
(complete=1, tag=node_state, id=d969970d-1439-4a47-bf60-3b8d3fc4558b,
magic=NA, cib=0.15.3) : Node failure
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no
mbr_track info
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: mem_handle_event:
instance=8, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=5
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback:
Quorum lost after event=INVALID (id=8)
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
INVALID: trans=8, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=5
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
#011CURRENT: vsaas-local-db01 [nodeid=0, born=8]
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
#011LOST:    vsaas-local-ws01 [nodeid=2, born=6]
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: ais_status_callback:
status: vsaas-local-ws01 is now lost (was member)
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no
mbr_track info
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: crm_update_peer: Node
vsaas-local-ws01: id=2 state=lost (new) addr=(null) votes=-1 born=6 seen=7
proc=00000000000000000000000000000200
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: erase_node_from_join:
Removed node vsaas-local-ws01 from join calculations: welcomed=0
itegrated=0 finalized=0 confirmed=1
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: mem_handle_event:
instance=8, nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=5
Dec 19 05:33:06 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha:
Requesting the list of configured nodes
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback:
Processing CCM event=INVALID (id=8)
Dec 19 05:33:06 vsaas-local-db01 cib: [3183]: info: crm_update_peer: Node
vsaas-local-ws01: id=2 state=lost (new) addr=(null) votes=-1 born=6 seen=7
proc=00000000000000000000000000000302
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
All 1 cluster nodes are eligible to run resources.
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query
58: Requesting the current CIB: S_POLICY_ENGINE
Dec 19 05:33:08 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/56,
version=0.15.3): ok (rc=0)
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke_callback:
Invoking the PE: query=58, ref=pe_calc-dc-1324301588-36, seq=8, quorate=0
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: unpack_config: On
loss of CCM Quorum: Ignore
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: unpack_config: Node
scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info:
determine_online_status: Node vsaas-local-db01 is online
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: native_print:
sql-master-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: native_print:
sql-slave-ip#011(ocf::heartbeat:IPaddr):#011Started vsaas-local-db01
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: colocation_match:
sql-master-ip: Rolling back scores from sql-slave-ip (1, -100)
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
resource sql-master-ip#011(Started vsaas-local-db01)
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: notice: LogActions: Leave
resource sql-slave-ip#011(Started vsaas-local-db01)
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: unpack_graph: Unpacked
transition 2: 0 actions in 0 synapses
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_te_invoke:
Processing graph 2 (ref=pe_calc-dc-1324301588-36) derived from
/var/lib/pengine/pe-input-2.bz2
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: run_graph:
====================================================
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: notice: run_graph:
Transition 2 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-2.bz2): Complete
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: te_graph_trigger:
Transition 2 is now complete
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: notify_crmd:
Transition 2 status: done - <null>
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 19 05:33:08 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
Starting PEngine Recheck Timer
Dec 19 05:33:08 vsaas-local-db01 pengine: [3557]: info: process_pe_message:
Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-2.bz2
Dec 19 05:34:11 vsaas-local-db01 heartbeat: [3167]: info: Link
vsaas-local-db02:eth0 up.
Dec 19 05:34:11 vsaas-local-db01 cib: [3183]: WARN: cib_peer_callback:
Discarding cib_apply_diff message (8b59) from vsaas-local-db02: not in our
membership
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: fromnode
=vsaas-local-db02, fromnode's ackseq = 0
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hist information:
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: hiseq =35686,
lowseq=35643,ackseq=35643,lastmsg=185
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: For information
on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: WARN: Deadtime value
may be too small.
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: See FAQ for
information on tuning deadtime.
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: URL:
http://linux-ha.org/wiki/FAQ#Heavy_Load
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: WARN: Late heartbeat:
Node vsaas-local-db02: interval 122790 ms
Dec 19 05:34:12 vsaas-local-db01 heartbeat: [3167]: info: Status update for
node vsaas-local-db02: status active
Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: notice:
crmd_ha_status_callback: Status update: Node vsaas-local-db02 now has
status [active] (DC=true)
Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: info: crm_update_peer_proc:
vsaas-local-db02.ais is now online
Dec 19 05:34:12 vsaas-local-db01 crmd: [3187]: WARN: crmd_ha_msg_callback:
Ignoring HA message (op=vote) from vsaas-local-db02: not in our membership
list (size=1)
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no
mbr_track info
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: mem_handle_event:
instance=9, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=5
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback:
Quorum lost after event=INVALID (id=9)
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
INVALID: trans=9, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=5
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
#011CURRENT: vsaas-local-db01 [nodeid=0, born=9]
Dec 19 05:34:13 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha:
Requesting the list of configured nodes
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no
mbr_track info
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: mem_handle_event:
instance=9, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=5
Dec 19 05:34:13 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback:
Processing CCM event=INVALID (id=9)
Dec 19 05:34:15 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/60,
version=0.15.5): ok (rc=0)
Dec 19 05:34:15 vsaas-local-db01 cib: [3183]: WARN: cib_peer_callback:
Discarding cib_apply_diff message (8b62) from vsaas-local-db02: not in our
membership
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: no
mbr_track info
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: mem_handle_event:
instance=10, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=6
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: crmd_ccm_msg_callback:
Quorum lost after event=INVALID (id=10)
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
INVALID: trans=10, nodes=2, new=1, lost=0 n_idx=0, new_idx=2, old_idx=6
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
#011CURRENT: vsaas-local-db01 [nodeid=0, born=1]
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
#011CURRENT: vsaas-local-db02 [nodeid=1, born=10]
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ccm_event_detail:
#011NEW:     vsaas-local-db02 [nodeid=1, born=10]
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: ais_status_callback:
status: vsaas-local-db02 is now member (was lost)
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: crm_update_peer: Node
vsaas-local-db02: id=1 state=member (new) addr=(null) votes=-1 born=10
seen=10 proc=00000000000000000000000000000202
Dec 19 05:34:16 vsaas-local-db01 crmd: [3187]: info: populate_cib_nodes_ha:
Requesting the list of configured nodes
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: no
mbr_track info
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event: Got
an event OC_EV_MS_INVALID from ccm
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: mem_handle_event:
instance=10, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=6
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_ccm_msg_callback:
Processing CCM event=INVALID (id=10)
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: crm_update_peer: Node
vsaas-local-db02: id=1 state=member (new) addr=(null) votes=-1 born=10
seen=10 proc=00000000000000000000000000000302
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='vsaas-local-db02']/lrm (origin=local/crmd/62,
version=0.15.6): ok (rc=0)
Dec 19 05:34:16 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='vsaas-local-db02']/transient_attributes
(origin=local/crmd/63, version=0.15.7): ok (rc=0)
Dec 19 05:34:17 vsaas-local-db01 cib: [3183]: WARN: cib_process_diff: Diff
0.14.20 -> 0.15.1 not applied to 0.15.7: current "epoch" is greater than
required
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info:
abort_transition_graph: te_update_diff:267 - Triggered transition abort
(complete=1, tag=lrm_rsc_op, id=sql-master-ip_monitor_0,
magic=0:7;4:3:7:e132d41b-0185-4355-9cbc-e0acb43058c5, cib=0.15.6) :
Resource op removal
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='vsaas-local-db02']/lrm": ok (rc=0)
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info:
abort_transition_graph: te_update_diff:157 - Triggered transition abort
(complete=1, tag=transient_attributes,
id=f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1, magic=NA, cib=0.15.7) : Transient
attribute: removal
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='vsaas-local-db02']/transient_attributes":
ok (rc=0)
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
Membership changed: 7 -> 10 - join restart
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_pe_invoke: Query
66: Requesting the current CIB: S_POLICY_ENGINE
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=do_state_transition ]
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
vsaas-local-db01
Dec 19 05:34:17 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section nodes (origin=local/crmd/64,
version=0.15.7): ok (rc=0)
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: join_make_offer:
Making join offers based on membership 10
Dec 19 05:34:17 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-3: Waiting on 2 outstanding join acks
Dec 19 05:34:18 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
vsaas-local-db01 (3.0.1)
Dec 19 05:34:19 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:34:19 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
vsaas-local-db01
Dec 19 05:34:19 vsaas-local-db01 cib: [3183]: WARN: cib_process_diff: Diff
0.15.1 -> 0.15.2 not applied to 0.15.8: current "num_updates" is greater
than required
Dec 19 05:36:15 vsaas-local-db01 crmd: [3187]: info: crmd_ha_msg_filter:
Another DC detected: vsaas-local-db02 (op=join_offer)
Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:36:19 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/68, version=0.15.8): ok (rc=0)
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/69,
version=0.15.8): ok (rc=0)
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/70,
version=0.15.8): ok (rc=0)
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/72, version=0.15.8): ok (rc=0)
Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-4: Waiting on 2 outstanding join acks
Dec 19 05:36:22 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/74, version=0.15.8): ok (rc=0)
Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:36:22 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
vsaas-local-db01 (3.0.1)
Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 5 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Dec 19 05:36:23 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
vsaas-local-db01
Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:38:23 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/77, version=0.15.8): ok (rc=0)
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/78,
version=0.15.8): ok (rc=0)
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/79,
version=0.15.8): ok (rc=0)
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/81, version=0.15.8): ok (rc=0)
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-5: Waiting on 2 outstanding join acks
Dec 19 05:38:26 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/83, version=0.15.8): ok (rc=0)
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:38:26 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
I_JOIN_OFFER from route_message() received in state S_ELECTION
Dec 19 05:38:27 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 6 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:40:26 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/85, version=0.15.8): ok (rc=0)
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/86,
version=0.15.8): ok (rc=0)
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/87,
version=0.15.8): ok (rc=0)
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/89, version=0.15.8): ok (rc=0)
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-6: Waiting on 2 outstanding join acks
Dec 19 05:40:29 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/91, version=0.15.8): ok (rc=0)
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:40:29 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
I_JOIN_OFFER from route_message() received in state S_ELECTION
Dec 19 05:40:30 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 7 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:42:29 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/93, version=0.15.8): ok (rc=0)
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/94,
version=0.15.8): ok (rc=0)
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/95,
version=0.15.8): ok (rc=0)
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/97, version=0.15.8): ok (rc=0)
Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-7: Waiting on 2 outstanding join acks
Dec 19 05:42:32 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/99, version=0.15.8): ok (rc=0)
Dec 19 05:42:32 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
vsaas-local-db01 (3.0.1)
Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:42:33 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
vsaas-local-db01
Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 8 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 9 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:42:34 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
I_JOIN_REQUEST from route_message() received in state S_ELECTION
Dec 19 05:43:03 vsaas-local-db01 cib: [3183]: info: cib_stats: Processed 54
operations (2962.00us average, 0% utilization) in the last 10min
Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:44:33 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/102, version=0.15.8): ok (rc=0)
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/103,
version=0.15.8): ok (rc=0)
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/104,
version=0.15.8): ok (rc=0)
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/106, version=0.15.8): ok (rc=0)
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-8: Waiting on 2 outstanding join acks
Dec 19 05:44:36 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/108, version=0.15.8): ok (rc=0)
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:44:36 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
I_JOIN_OFFER from route_message() received in state S_ELECTION
Dec 19 05:44:38 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 10 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:46:28 vsaas-local-db01 heartbeat: [3167]: info: killing
/usr/lib/heartbeat/crmd process group 3187 with signal 15
Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: crm_shutdown:
Requesting shutdown
Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info: do_shutdown_req:
Sending shutdown request to DC: <null>
Dec 19 05:46:28 vsaas-local-db01 crmd: [3187]: info:
handle_shutdown_request: Creating shutdown request for vsaas-local-db01
(state=S_ELECTION)
Dec 19 05:46:28 vsaas-local-db01 attrd: [3186]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown (1324302388)
Dec 19 05:46:28 vsaas-local-db01 attrd: [3186]: info: attrd_perform_update:
Sent update 97: shutdown=1324302388
Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:46:36 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/111, version=0.15.10): ok (rc=0)
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/112,
version=0.15.10): ok (rc=0)
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/113,
version=0.15.10): ok (rc=0)
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/115, version=0.15.10): ok (rc=0)
Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-9: Waiting on 2 outstanding join acks
Dec 19 05:46:39 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/117, version=0.15.10): ok (rc=0)
Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:46:39 vsaas-local-db01 crmd: [3187]: info: update_dc: Set DC to
vsaas-local-db01 (3.0.1)
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info: update_dc: Unset DC
vsaas-local-db01
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 11 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 12 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
Dec 19 05:46:40 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
I_JOIN_REQUEST from route_message() received in state S_ELECTION
Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: info: start_subsystem:
Starting sub-system "pengine"
Dec 19 05:48:40 vsaas-local-db01 crmd: [3187]: WARN: start_subsystem:
Client pengine already running as pid 3557
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_dc_takeover: Taking
over DC status for this partition
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/O mode
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_slave_all for section 'all'
(origin=local/crmd/120, version=0.15.10): ok (rc=0)
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_readwrite:
We are now in R/W mode
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_master for section 'all' (origin=local/crmd/121,
version=0.15.10): ok (rc=0)
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section cib (origin=local/crmd/122,
version=0.15.10): ok (rc=0)
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/124, version=0.15.10): ok (rc=0)
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_dc_join_offer_all:
join-10: Waiting on 2 outstanding join acks
Dec 19 05:48:43 vsaas-local-db01 cib: [3183]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/126, version=0.15.10): ok (rc=0)
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: config_query_callback:
Checking for expired actions every 900000ms
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Dec 19 05:48:43 vsaas-local-db01 crmd: [3187]: WARN: do_log: FSA: Input
I_JOIN_OFFER from route_message() received in state S_ELECTION
Dec 19 05:48:44 vsaas-local-db01 crmd: [3187]: info:
do_election_count_vote: Election 13 (owner:
f6bdf2e7-9124-48a0-a0b7-be293c6eb5e1) pass: vote from vsaas-local-db02 (Age)
root at vsaas-local-db01:/home/mikema#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20111219/915bdeab/attachment-0001.html>