Mar 24 10:56:15 ha-test2 cib: [4823]: info: cib_stats: Processed 128 operations (234.00us average, 0% utilization) in the last 10min
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 48: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=48, ref=pe_calc-dc-1269424941-53, seq=840, quorate=0
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 4: 0 actions in 0 synapses
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1269424941-53) derived from /var/lib/pengine/pe-input-332.bz2
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:02:21 ha-test1 crmd: [9046]: notice: run_graph: Transition 4 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-332.bz2): Complete
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 4 is now complete
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: notify_crmd: Transition 4 status: done - <null>
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/pengine/pe-input-332.bz2
Mar 24 11:06:12 ha-test1 cib: [9042]: info: cib_stats: Processed 33 operations (606.00us average, 0% utilization) in the last 10min
Mar 24 11:06:15 ha-test2 cib: [4823]: info: cib_stats: Processed 32 operations (625.00us average, 0% utilization) in the last 10min
Mar 24 11:16:12 ha-test1 cib: [9042]: info: cib_stats: Processed 32 operations (312.00us average, 0% utilization) in the last 10min
Mar 24 11:16:15 ha-test2 cib: [4823]: info: cib_stats: Processed 32 operations (0.00us average, 0% utilization) in the last 10min
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 49: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=49, ref=pe_calc-dc-1269425841-54, seq=840, quorate=0
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 5: 0 actions in 0 synapses
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1269425841-54) derived from /var/lib/pengine/pe-input-333.bz2
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:17:21 ha-test1 crmd: [9046]: notice: run_graph: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-333.bz2): Complete
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 5 is now complete
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: notify_crmd: Transition 5 status: done - <null>
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-333.bz2
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="130" num_updates="42" >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="131" num_updates="1" >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 50: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.131.1): ok (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=50, ref=pe_calc-dc-1269425968-55, seq=840, quorate=0
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Demote resDRBD:0#011(Master -> Stopped ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDRBD:0#011(ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDLM:1#011(ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resO2CB:1#011(ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resFS:1#011(ha-test1)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:28 ha-test1 cib: [14490]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-69.raw
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 6: 31 actions in 31 synapses
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1269425968-55) derived from /var/lib/pengine/pe-warn-459.bz2
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 39 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 62 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 80: notify resDRBD:0_pre_notify_demote_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=80:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:20: notify
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 82: notify resDRBD:1_pre_notify_demote_0 on ha-test2
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: stop resFS:1_stop_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=59:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_stop_0 )
Mar 24 11:19:28 ha-test1 cib: [14490]: info: write_cib_contents: Wrote version 0.131.0 of the CIB to disk (digest: 0ba795530b6b6d0f945897bcb70a2107)
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resFS:1:21: stop
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=19, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 6: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-459.bz2
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:19:28 ha-test1 cib: [14490]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.9jHvtg (digest: /var/lib/heartbeat/crm/cib.Bnb1E3)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_demote_0 (82) confirmed on ha-test2 (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=20, rc=0, cib-update=51, confirmed=true) ok
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_demote_0 (80) confirmed on ha-test1 (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 40 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: demote resDRBD:0_demote_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_demote_0 )
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:22: demote
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=14, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:28 ha-test1 Filesystem[14492]: INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:19:28 INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:19:28 ha-test1 Filesystem[14492]: INFO: Trying to unmount /nfs
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:19:28 INFO: Trying to unmount /nfs
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0: State change failed: (-12) Device is held open by someone
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) Command '
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) drbdsetup
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) secondary
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) ' terminated with exit code 11
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Command output: 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Command output: 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:19:28 ha-test1 Filesystem[14492]: INFO: unmounted /nfs successfully
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:19:28 INFO: unmounted /nfs successfully
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) /dev/drbd0: Wrong medium type
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_stop_0 (call=21, rc=0, cib-update=52, confirmed=true) ok
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_stop_0 (59) confirmed on ha-test1 (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: stop resO2CB:1_stop_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=52:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_stop_0 )
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:23: stop
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=17, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:28 ha-test2 crm_shadow: [10214]: info: Invoked: crm_shadow 
Mar 24 11:19:28 ha-test2 crm_attribute: [10219]: info: Invoked: crm_attribute -N ha-test1 -n standby -v on -l forever 
Mar 24 11:19:28 ha-test2 cib: [10220]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-47.raw
Mar 24 11:19:28 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=82:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:28 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:20: notify
Mar 24 11:19:28 ha-test2 cib: [10220]: info: write_cib_contents: Wrote version 0.131.0 of the CIB to disk (digest: 21d74a7a136d837dc73311b025e9f44f)
Mar 24 11:19:28 ha-test2 cib: [10220]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.yGKN9r (digest: /var/lib/heartbeat/crm/cib.NKBdFL)
Mar 24 11:19:28 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=20, rc=0, cib-update=27, confirmed=true) ok
Mar 24 11:19:29 ha-test1 o2cb[14635]: INFO: Stopping resO2CB:1
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:19:29 INFO: Stopping resO2CB:1
Mar 24 11:19:29 ha-test1 o2cb[14635]: INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:19:29 INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) /usr/lib/ocf/resource.d//pacemaker/o2cb: line 169: killproc: command not found
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_stop_0 (call=23, rc=0, cib-update=53, confirmed=true) ok
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_stop_0 (52) confirmed on ha-test1 (rc=0)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: stop resDLM:1_stop_0 on ha-test1 (local)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_stop_0 )
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:24: stop
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=15, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:29 ha-test1 ocfs2_controld[9480]: Error from dlmc_fs_result: Bad file descriptor
Mar 24 11:19:29 ha-test1 ocfs2_controld[9480]: dlmcontrol connection died
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:stop:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_demote_0 (call=22, rc=0, cib-update=54, confirmed=true) ok
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_demote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 38 fired and confirmed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 41 fired and confirmed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:0_post_notify_demote_0 on ha-test1 (local)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=81:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:25: notify
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 83: notify resDRBD:1_post_notify_demote_0 on ha-test2
Mar 24 11:19:29 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=83:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:29 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:21: notify
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=25, rc=0, cib-update=55, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_demote_0 (81) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_demote_0 (83) confirmed on ha-test2 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 42 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 75: notify resDRBD:0_pre_notify_stop_0 on ha-test1 (local)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=75:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:26: notify
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 76: notify resDRBD:1_pre_notify_stop_0 on ha-test2
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=26, rc=0, cib-update=56, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_stop_0 (75) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_stop_0 (call=24, rc=0, cib-update=57, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_stop_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 49 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_stop_0 (76) confirmed on ha-test2 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 25 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 14: stop resDRBD:0_stop_0 on ha-test1 (local)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=14:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_stop_0 )
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:27: stop
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (-INFINITY)
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 19: master-resDRBD:0=-INFINITY
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.131.12) : Transient attribute: update
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 crm_attribute: [14780]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent delete 21: node=ha-test1, attr=master-resDRBD:0, id=<n/a>, set=(null), section=status
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.131.13) : Transient attribute: removal
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_stop_0 (call=27, rc=0, cib-update=58, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_stop_0 (14) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 77: notify resDRBD:1_post_notify_stop_0 on ha-test2
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_stop_0 (77) confirmed on ha-test2 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 30 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:30 ha-test1 crmd: [9046]: notice: run_graph: Transition 6 (Complete=30, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-warn-459.bz2): Stopped
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 6 is now complete
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 59: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=59, ref=pe_calc-dc-1269425970-68, seq=840, quorate=0
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Stopped)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 7: 0 actions in 0 synapses
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1269425970-68) derived from /var/lib/pengine/pe-warn-460.bz2
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:30 ha-test1 crmd: [9046]: notice: run_graph: Transition 7 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-460.bz2): Complete
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 7 is now complete
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: notify_crmd: Transition 7 status: done - <null>
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 7: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-460.bz2
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=21, rc=0, cib-update=28, confirmed=true) ok
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=76:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:22: notify
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=22, rc=0, cib-update=29, confirmed=true) ok
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=77:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:23: notify
Mar 24 11:19:30 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:19:30 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=23, rc=0, cib-update=30, confirmed=true) ok
Mar 24 11:19:36 ha-test2 crm_shadow: [10349]: info: Invoked: crm_shadow 
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="131" num_updates="15" >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="132" num_updates="1" >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 60: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.132.1): ok (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=60, ref=pe_calc-dc-1269425978-69, seq=840, quorate=0
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:0#011(ha-test1)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 8: 11 actions in 11 synapses
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1269425978-69) derived from /var/lib/pengine/pe-warn-461.bz2
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 70: notify resDRBD:1_pre_notify_start_0 on ha-test2
Mar 24 11:19:38 ha-test1 cib: [14781]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-70.raw
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 8: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-461.bz2
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:19:38 ha-test1 cib: [14781]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: c9bf29eac5472da64c039e3089cea654)
Mar 24 11:19:38 ha-test1 cib: [14781]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.H3ovPO (digest: /var/lib/heartbeat/crm/cib.BBA0ma)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_start_0 (70) confirmed on ha-test2 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: start resDRBD:0_start_0 on ha-test1 (local)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=9:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_start_0 )
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:28: start
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout)         allow-two-primaries;
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:19:38 ha-test1 crm_attribute: [14834]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_start_0 (call=28, rc=0, cib-update=61, confirmed=true) ok
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_start_0 (9) confirmed on ha-test1 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 69: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=69:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:29: notify
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 71: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=29, rc=0, cib-update=62, confirmed=true) ok
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (69) confirmed on ha-test1 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (71) confirmed on ha-test2 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=10:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_30000 )
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:30: monitor
Mar 24 11:19:38 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10)
Mar 24 11:19:38 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 25: master-resDRBD:0=10
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.132.6) : Transient attribute: update
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:19:38 ha-test2 crm_shadow: [10379]: info: Invoked: crm_shadow 
Mar 24 11:19:38 ha-test2 crm_attribute: [10383]: info: Invoked: crm_attribute -N ha-test1 -n standby -v off -l forever 
Mar 24 11:19:38 ha-test2 cib: [10384]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-48.raw
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=70:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:24: notify
Mar 24 11:19:38 ha-test2 cib: [10384]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: c431886c0decf8058af7150f21a06f18)
Mar 24 11:19:38 ha-test2 cib: [10384]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.aUCFt0 (digest: /var/lib/heartbeat/crm/cib.F5gXiS)
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=24, rc=0, cib-update=31, confirmed=true) ok
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=71:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:25: notify
Mar 24 11:19:38 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:19:38 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=25, rc=0, cib-update=32, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=30, rc=0, cib-update=63, confirmed=false) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (10) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:39 ha-test1 crmd: [9046]: notice: run_graph: Transition 8 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-461.bz2): Complete
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 8 is now complete
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 64: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=64, ref=pe_calc-dc-1269425979-75, seq=840, quorate=0
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:0#011(Slave -> Master ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:1#011(ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 9: 25 actions in 25 synapses
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 9 (ref=pe_calc-dc-1269425979-75) derived from /var/lib/pengine/pe-input-334.bz2
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 1: cancel resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: send_direct_ack: ACK'ing resource op resDRBD:0_monitor_30000 from 1:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269425979-77
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269425979-77 from ha-test1
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (1) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=30, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=81:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:31: notify
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 83: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: start resDLM:1_start_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_start_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:32: start
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: process_pe_message: Transition 9: PEngine Input stored in: /var/lib/pengine/pe-input-334.bz2
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (83) confirmed on ha-test2 (rc=0)
Mar 24 11:19:39 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 14931 (0x885f638)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=31, rc=0, cib-update=66, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (81) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: promote resDRBD:0_promote_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_promote_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:33: promote
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout)         allow-two-primaries;
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout) 
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_promote_0 (call=33, rc=0, cib-update=67, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_promote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 82: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=82:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:34: notify
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 84: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (84) confirmed on ha-test2 (rc=0)
Mar 24 11:19:39 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:19:39 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 27: master-resDRBD:0=10000
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.132.13) : Transient attribute: update
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=34, rc=0, cib-update=68, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (82) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=83:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:39 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:26: notify
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=26, rc=0, cib-update=33, confirmed=true) ok
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: do_lrm_rsc_op: Performing key=84:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:19:39 ha-test2 lrmd: [4824]: info: rsc:resDRBD:1:27: notify
Mar 24 11:19:39 ha-test2 lrmd: [4824]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:19:39 ha-test2 crmd: [4827]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=27, rc=0, cib-update=34, confirmed=true) ok
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_start_0 (call=32, rc=0, cib-update=69, confirmed=true) ok
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_start_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:40 ha-test1 crmd: [9046]: notice: run_graph: Transition 9 (Complete=17, Pending=0, Fired=0, Skipped=6, Incomplete=2, Source=/var/lib/pengine/pe-input-334.bz2): Stopped
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 9 is now complete
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 70: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=70, ref=pe_calc-dc-1269425980-84, seq=840, quorate=0
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:40 ha-test1 crmd: [9046]: WARN: destroy_action: Cancelling timer for action 1 (src=129)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 10: 10 actions in 10 synapses
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 10 (ref=pe_calc-dc-1269425980-84) derived from /var/lib/pengine/pe-input-335.bz2
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: monitor resDRBD:0_monitor_20000 on ha-test1 (local)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:10:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_20000 )
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:35: monitor
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 46: monitor resDLM:1_monitor_120000 on ha-test1 (local)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=46:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_monitor_120000 )
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:36: monitor
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 53: start resO2CB:1_start_0 on ha-test1 (local)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=53:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_start_0 )
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:37: start
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: process_pe_message: Transition 10: PEngine Input stored in: /var/lib/pengine/pe-input-335.bz2
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=36, rc=0, cib-update=71, confirmed=false) ok
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_monitor_120000 (46) confirmed on ha-test1 (rc=0)
Mar 24 11:19:40 ha-test1 o2cb[15017]: INFO: ocfs2 not loaded
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:19:40 INFO: ocfs2 not loaded
Mar 24 11:19:40 ha-test1 o2cb[15017]: INFO: Starting resO2CB:1
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:19:40 INFO: Starting resO2CB:1
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=35, rc=8, cib-update=72, confirmed=false) master
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_20000 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:19:40 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 15089 (0x885a160)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_start_0 (call=37, rc=0, cib-update=73, confirmed=true) ok
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_start_0 (53) confirmed on ha-test1 (rc=0)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 54: monitor resO2CB:1_monitor_120000 on ha-test1 (local)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=54:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_monitor_120000 )
Mar 24 11:19:42 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:38: monitor
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 61: start resFS:1_start_0 on ha-test1 (local)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=61:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_start_0 )
Mar 24 11:19:42 ha-test1 lrmd: [9043]: info: rsc:resFS:1:39: start
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=38, rc=0, cib-update=74, confirmed=false) ok
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_monitor_120000 (54) confirmed on ha-test1 (rc=0)
Mar 24 11:19:42 ha-test1 Filesystem[15102]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:19:42 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:start:stderr) 2010/03/24_11:19:42 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_start_0 (call=39, rc=0, cib-update=75, confirmed=true) ok
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_start_0 (61) confirmed on ha-test1 (rc=0)
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 62: monitor resFS:1_monitor_120000 on ha-test1 (local)
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=62:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_monitor_120000 )
Mar 24 11:19:43 ha-test1 lrmd: [9043]: info: rsc:resFS:1:40: monitor
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=40, rc=0, cib-update=76, confirmed=false) ok
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_monitor_120000 (62) confirmed on ha-test1 (rc=0)
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:43 ha-test1 crmd: [9046]: notice: run_graph: Transition 10 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-335.bz2): Complete
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 10 is now complete
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: notify_crmd: Transition 10 status: done - <null>
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:22:42 ha-test1 corosync[9029]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:22:42 ha-test2 corosync[4811]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:22:42 ha-test2 corosync[4811]:   [TOTEM ] The network interface is down.
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 844: memb=1, new=0, lost=1
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: lost: ha-test2 1550025994
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 844: memb=1, new=0, lost=0
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: ais_mark_unseen_peer_dead: Node ha-test2 was not seen in the previous transition
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: lost
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 844 to 0 children
Mar 24 11:22:45 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:22:45 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:22:45 ha-test1 cib: [9042]: info: ais_dispatch: Membership 844: quorum still lost
Mar 24 11:22:45 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000348(null)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 844: quorum still lost
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now lost (was member)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000348(null)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: erase_node_from_join: Removed node ha-test2 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
Mar 24 11:22:45 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/77, version=0.132.21): ok (rc=0)
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="844" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:22:45 ha-test1 crmd: [9046]: WARN: match_down_event: No match for shutdown action on ha-test2
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: te_update_diff: Stonith/shutdown of ha-test2 not matched
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=ha-test2, magic=NA, cib=0.132.22) : Node failure
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 79: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=79, ref=pe_calc-dc-1269426165-92, seq=844, quorate=0
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:1 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 11: 0 actions in 0 synapses
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 11 (ref=pe_calc-dc-1269426165-92) derived from /var/lib/pengine/pe-warn-462.bz2
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:22:45 ha-test1 crmd: [9046]: notice: run_graph: Transition 11 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-462.bz2): Complete
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 11 is now complete
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: notify_crmd: Transition 11 status: done - <null>
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 11: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-462.bz2
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 848: memb=1, new=0, lost=0
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 848: memb=2, new=1, lost=0
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: member
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: NEW:  ha-test2 1550025994
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 848 to 0 children
Mar 24 11:23:12 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:23:12 ha-test1 cib: [9042]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now member (was lost)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/80, version=0.132.23): ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=local/crmd/81, version=0.132.24): ok (rc=0)
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: 0x88590d0 Node 1550025994 (ha-test2) born on: 848
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/82, version=0.132.24): ok (rc=0)
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 848 to 0 children
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 cib: [9042]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=resDLM:0_monitor_0, magic=0:7;10:1:7:ddac39ed-6665-4cec-98ce-67e0216b7011, cib=0.132.23) : Resource op removal
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.132.24) : Transient attribute: removal
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/84, version=0.132.25): ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_state_transition: Membership changed: 840 -> 848 - join restart
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 86: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: join_make_offer: Making join offers based on membership 848
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:23:12 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:23:12 ha-test2 cib: [1496]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: a6e332c373698e925ac0aa9d889cecf0)
Mar 24 11:23:12 ha-test2 cib: [1496]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.VJP89B (digest: /var/lib/heartbeat/crm/cib.anww6n)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_cib_control: CIB connection established
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: init_ais_connection: AIS connection established
Mar 24 11:23:13 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x8ac65d8 for crmd/1492
Mar 24 11:23:13 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_ipc: Sending membership update 848 to (null)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_ha_control: Connected to the cluster
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_started: Delaying start, CCM (0000000000100000) not connected
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crmd_init: Starting crmd's mainloop
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  (new) votes=1 (new) born=848 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node ha-test1 now has id: 1533248778
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_new_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: crm_update_peer: Node ha-test1: id=1533248778 state=member (new) addr=r(0) ip(10.133.99.91)  votes=1 born=840 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_started: Delaying start, Config not read (0000000000000040)
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: config_query_callback: Checking for expired actions every 900000ms
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_started: The local CRM is operational
Mar 24 11:23:13 ha-test2 corosync[1466]:   [pcmk  ] info: update_expected_votes: Expected quorum votes 1024 -> 2
Mar 24 11:23:13 ha-test2 crmd: [1492]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: A new node joined the cluster
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-3: Waiting on 2 outstanding join acks
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_finalize: join-3: Syncing the CIB from ha-test1 to the rest of the cluster
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/89, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/90, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/91, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=ha-test2/crmd/6, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=ha-test2/crmd/7, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-3: Updating node state to member for ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-3: Updating node state to member for ha-test1
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/92, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test1']/lrm (origin=local/crmd/94, version=0.132.27): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/96, version=0.132.28): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: crm_update_quorum: Updating quorum status to false (call=98)
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/98, version=0.132.28): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 99: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test1']/lrm": ok (rc=0)
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_update_diff: Detected LRM refresh - 7 resources updated: Skipping all resource events
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:227 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.132.28) : LRM Refresh
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (<null>)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=99, ref=pe_calc-dc-1269426194-104, seq=848, quorate=0
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 100: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=100, ref=pe_calc-dc-1269426194-105, seq=848, quorate=0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:1#011(ha-test2)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: handle_response: pe_calc calculation pe_calc-dc-1269426194-104 is obsolete
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 12: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-463.bz2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:1#011(ha-test2)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 13: 17 actions in 17 synapses
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 13 (ref=pe_calc-dc-1269426194-105) derived from /var/lib/pengine/pe-warn-464.bz2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: monitor resDRBD:1_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDLM:0_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 11: monitor resO2CB:0_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 12: monitor resFS:0_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 77: notify resDRBD:0_pre_notify_start_0 on ha-test1 (local)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=77:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:14 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:41: notify
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 13: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-464.bz2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:23:14 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=41, rc=0, cib-update=101, confirmed=true) ok
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_start_0 (77) confirmed on ha-test1 (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:23:14 ha-test2 cib: [1488]: info: cib_replace_notify: Replaced: 0.132.0 -> 0.132.25 from ha-test1
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: update_attrd: Connecting to attrd...
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for terminate
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for shutdown
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:23:14 ha-test2 logd: [1858]: info: logd started with default configuration.
Mar 24 11:23:14 ha-test2 logd: [1858]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:23:14 ha-test2 logd: [1862]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:23:14 ha-test2 cib: [1855]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-50.raw
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: crm_new_peer: Node ha-test1 now has id: 1533248778
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: crm_new_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for master-resDRBD:0
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:0=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for probe_complete
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation probe_complete=<null>: cib not connected
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: find_hash_entry: Creating hash entry for master-resDRBD:1
Mar 24 11:23:14 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:1=<null>: cib not connected
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=9:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:2: monitor
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=10:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resDLM:0:3: monitor
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=11:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resO2CB:0:4: monitor
Mar 24 11:23:14 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=12:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_0 )
Mar 24 11:23:14 ha-test2 lrmd: [1489]: info: rsc:resFS:0:5: monitor
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_0 (10) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_0 (11) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_0 (12) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_0 (9) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on ha-test2 - no waiting
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 6 fired and confirmed
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: start resDRBD:1_start_0 on ha-test2
Mar 24 11:23:15 ha-test2 Filesystem[1891]: WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resFS:0:monitor:stderr) 2010/03/24_11:23:15 WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resDLM:0:monitor:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDLM:0_monitor_0 (call=3, rc=7, cib-update=8, confirmed=true) not running
Mar 24 11:23:15 ha-test2 cib: [1855]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: 075394cb7bbc83fedfbbd40279948d19)
Mar 24 11:23:15 ha-test2 o2cb[1890]: INFO: configfs not laoded
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resO2CB:0:monitor:stderr) 2010/03/24_11:23:15 INFO: configfs not laoded
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resO2CB:0_monitor_0 (call=4, rc=7, cib-update=9, confirmed=true) not running
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resFS:0_monitor_0 (call=5, rc=7, cib-update=10, confirmed=true) not running
Mar 24 11:23:15 ha-test2 cib: [1855]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.qy4I1I (digest: /var/lib/heartbeat/crm/cib.4lRGPB)
Mar 24 11:23:15 ha-test2 crm_attribute: [2031]: info: Invoked: crm_attribute -N ha-test2 -n master-resDRBD:1 -l reboot -D 
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_0 (call=2, rc=7, cib-update=11, confirmed=true) not running
Mar 24 11:23:15 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:23:15 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation probe_complete=true: cib not connected
Mar 24 11:23:15 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=17:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_start_0 )
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:6: start
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout)         allow-two-primaries;
Mar 24 11:23:15 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_start_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 78: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=78:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:16 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:42: notify
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 79: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:23:16 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=42, rc=0, cib-update=102, confirmed=true) ok
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (78) confirmed on ha-test1 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (79) confirmed on ha-test2 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:23:16 ha-test1 crmd: [9046]: notice: run_graph: Transition 13 (Complete=17, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-464.bz2): Complete
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 13 is now complete
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: notify_crmd: Transition 13 status: done - <null>
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:23:16 ha-test2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.33.1/kernel/drivers/crypto/padlock-sha.ko): No such device
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (1000)
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:1=1000: cib not connected
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_start_0 (call=6, rc=0, cib-update=12, confirmed=true) ok
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=79:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:7: notify
Mar 24 11:23:16 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=7, rc=0, cib-update=13, confirmed=true) ok
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=18:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_30000 )
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:23:16 ha-test2 attrd: [1490]: info: attrd_perform_update: Delaying operation master-resDRBD:1=10000: cib not connected
Mar 24 11:23:16 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, rc=0, cib-update=14, confirmed=false) ok
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.132.38) : Transient attribute: update
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 103: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.132.39) : Transient attribute: update
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 104: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=104, ref=pe_calc-dc-1269426197-116, seq=848, quorate=0
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test2 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:1#011(Slave -> Master ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:0#011(ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:0#011(ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:0#011(ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 14: 25 actions in 25 synapses
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 14 (ref=pe_calc-dc-1269426197-116) derived from /var/lib/pengine/pe-input-336.bz2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 5: cancel resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 85: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=85:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:43: notify
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 87: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 43: start resDLM:0_start_0 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269426197-4 from ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (5) confirmed on ha-test2 (rc=0)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: process_pe_message: Transition 14: PEngine Input stored in: /var/lib/pengine/pe-input-336.bz2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=43, rc=0, cib-update=105, confirmed=true) ok
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (85) confirmed on ha-test1 (rc=0)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (87) confirmed on ha-test2 (rc=0)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: promote resDRBD:1_promote_0 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_promote_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 86: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=86:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:44: notify
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 88: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: cib_connect: Connected to the CIB after 1 signon attempts
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: cib_connect: Sending full refresh
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_perform_update: Sent update 4: probe_complete=true
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:23:17 ha-test2 attrd: [1490]: info: attrd_perform_update: Sent update 10: master-resDRBD:1=10000
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: send_direct_ack: ACK'ing resource op resDRBD:1_monitor_30000 from 5:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269426197-4
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=87:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:9: notify
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=43:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_start_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDLM:0:10: start
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: RA output: (resDLM:0:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=9, rc=0, cib-update=16, confirmed=true) ok
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=17:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_promote_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:11: promote
Mar 24 11:23:17 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 2711 (0x8abfd60)
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:promote:stdout)         allow-two-primaries;
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:promote:stdout) 
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_promote_0 (call=11, rc=0, cib-update=17, confirmed=true) ok
Mar 24 11:23:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=88:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:23:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:12: notify
Mar 24 11:23:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=44, rc=0, cib-update=106, confirmed=true) ok
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (86) confirmed on ha-test1 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (88) confirmed on ha-test2 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_20000 on ha-test2
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_20000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_start_0 (43) confirmed on ha-test2 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 44: monitor resDLM:0_monitor_120000 on ha-test2
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 51: start resO2CB:0_start_0 on ha-test2
Mar 24 11:23:18 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=12, rc=0, cib-update=18, confirmed=true) ok
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=18:14:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_20000 )
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_monitor_20000 (call=13, rc=8, cib-update=19, confirmed=false) master
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDLM:0_start_0 (call=10, rc=0, cib-update=20, confirmed=true) ok
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=44:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_120000 )
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=51:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_start_0 )
Mar 24 11:23:18 ha-test2 lrmd: [1489]: info: rsc:resO2CB:0:15: start
Mar 24 11:23:18 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDLM:0_monitor_120000 (call=14, rc=0, cib-update=21, confirmed=false) ok
Mar 24 11:23:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_120000 (44) confirmed on ha-test2 (rc=0)
Mar 24 11:23:19 ha-test2 o2cb[3091]: INFO: Stack glue driver not loaded
Mar 24 11:23:19 ha-test2 lrmd: [1489]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:23:19 INFO: Stack glue driver not loaded
Mar 24 11:23:19 ha-test2 o2cb[3091]: INFO: Starting resO2CB:0
Mar 24 11:23:19 ha-test2 lrmd: [1489]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:23:19 INFO: Starting resO2CB:0
Mar 24 11:23:19 ha-test2 corosync[1466]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 3160 (0x8ac0fb8)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_start_0 (51) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: monitor resO2CB:0_monitor_120000 on ha-test2
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: start resFS:0_start_0 on ha-test2
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_120000 (52) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_start_0 (59) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 60: monitor resFS:0_monitor_120000 on ha-test2
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resO2CB:0_start_0 (call=15, rc=0, cib-update=22, confirmed=true) ok
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=52:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_120000 )
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=59:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_start_0 )
Mar 24 11:23:21 ha-test2 lrmd: [1489]: info: rsc:resFS:0:17: start
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resO2CB:0_monitor_120000 (call=16, rc=0, cib-update=23, confirmed=false) ok
Mar 24 11:23:21 ha-test2 Filesystem[3811]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:23:21 ha-test2 lrmd: [1489]: info: RA output: (resFS:0:start:stderr) 2010/03/24_11:23:21 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resFS:0_start_0 (call=17, rc=0, cib-update=24, confirmed=true) ok
Mar 24 11:23:21 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=60:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_120000 )
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_120000 (60) confirmed on ha-test2 (rc=0)
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:23:22 ha-test1 crmd: [9046]: notice: run_graph: Transition 14 (Complete=25, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-336.bz2): Complete
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 14 is now complete
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: notify_crmd: Transition 14 status: done - <null>
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:23:22 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resFS:0_monitor_120000 (call=18, rc=0, cib-update=25, confirmed=false) ok
Mar 24 11:25:22 ha-test2 cibadmin: [4425]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:25:22 ha-test2 root: Mark:HB_REPORT:1269426320
Mar 24 11:25:25 ha-test2 cibadmin: [4632]: info: Invoked: cibadmin -Ql 
Mar 24 11:25:25 ha-test2 crm_shadow: [4672]: info: Invoked: crm_shadow 
Mar 24 11:25:25 ha-test2 cibadmin: [4673]: info: Invoked: cibadmin -Ql 
Mar 24 11:26:12 ha-test1 cib: [9042]: info: cib_stats: Processed 202 operations (1633.00us average, 0% utilization) in the last 10min
Mar 24 11:28:54 ha-test2 cibadmin: [5558]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:28:54 ha-test2 root: Mark:HB_REPORT:1269426533
Mar 24 11:28:56 ha-test2 cibadmin: [5761]: info: Invoked: cibadmin -Ql 
Mar 24 11:28:56 ha-test2 crm_shadow: [5801]: info: Invoked: crm_shadow 
Mar 24 11:28:56 ha-test2 cibadmin: [5802]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:25 ha-test2 cibadmin: [6385]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:29:27 ha-test1 logger: Mark:HB_REPORT:1269426564
Mar 24 11:29:27 ha-test2 root: Mark:HB_REPORT:1269426564
Mar 24 11:29:29 ha-test2 cibadmin: [6579]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:29 ha-test2 crm_shadow: [6619]: info: Invoked: crm_shadow 
Mar 24 11:29:29 ha-test2 cibadmin: [6620]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:30 ha-test1 cibadmin: [16788]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:30 ha-test1 crm_shadow: [16828]: info: Invoked: crm_shadow 
Mar 24 11:29:30 ha-test1 cibadmin: [16829]: info: Invoked: cibadmin -Ql 
Mar 24 11:33:12 ha-test2 cib: [1488]: info: cib_stats: Processed 122 operations (573.00us average, 0% utilization) in the last 10min
Mar 24 11:35:22 ha-test2 cibadmin: [7890]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:35:26 ha-test1 logger: Mark:HB_REPORT:1269426925
Mar 24 11:35:26 ha-test2 cibadmin: [7970]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:35:26 ha-test2 root: Mark:HB_REPORT:1269426925
Mar 24 11:35:28 ha-test2 cibadmin: [8265]: info: Invoked: cibadmin -Ql 
Mar 24 11:35:28 ha-test2 crm_shadow: [8305]: info: Invoked: crm_shadow 
Mar 24 11:35:28 ha-test2 cibadmin: [8306]: info: Invoked: cibadmin -Ql 
Mar 24 11:35:29 ha-test1 cibadmin: [18451]: info: Invoked: cibadmin -Ql 
Mar 24 11:35:29 ha-test1 crm_shadow: [18491]: info: Invoked: crm_shadow 
Mar 24 11:35:30 ha-test1 cibadmin: [18492]: info: Invoked: cibadmin -Ql 
Mar 24 11:36:12 ha-test1 cib: [9042]: info: cib_stats: Processed 40 operations (2000.00us average, 0% utilization) in the last 10min
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 107: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=107, ref=pe_calc-dc-1269427102-130, seq=848, quorate=0
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 15: 0 actions in 0 synapses
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 15 (ref=pe_calc-dc-1269427102-130) derived from /var/lib/pengine/pe-input-337.bz2
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:38:22 ha-test1 crmd: [9046]: notice: run_graph: Transition 15 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-337.bz2): Complete
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 15 is now complete
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: notify_crmd: Transition 15 status: done - <null>
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: process_pe_message: Transition 15: PEngine Input stored in: /var/lib/pengine/pe-input-337.bz2
Mar 24 11:38:42 ha-test1 logger: Mark:HB_REPORT:1269427121
Mar 24 11:38:42 ha-test2 cibadmin: [9253]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:38:42 ha-test2 root: Mark:HB_REPORT:1269427121
Mar 24 11:38:44 ha-test2 cibadmin: [9548]: info: Invoked: cibadmin -Ql 
Mar 24 11:38:45 ha-test1 cibadmin: [19775]: info: Invoked: cibadmin -Ql 
Mar 24 11:38:45 ha-test2 crm_shadow: [9588]: info: Invoked: crm_shadow 
Mar 24 11:38:45 ha-test2 cibadmin: [9589]: info: Invoked: cibadmin -Ql 
Mar 24 11:38:46 ha-test1 crm_shadow: [19815]: info: Invoked: crm_shadow 
Mar 24 11:38:46 ha-test1 cibadmin: [19816]: info: Invoked: cibadmin -Ql 
Mar 24 11:43:12 ha-test2 cib: [1488]: info: cib_stats: Processed 46 operations (869.00us average, 0% utilization) in the last 10min
Mar 24 11:46:12 ha-test1 cib: [9042]: info: cib_stats: Processed 37 operations (1081.00us average, 0% utilization) in the last 10min
Mar 24 11:53:12 ha-test2 cib: [1488]: info: cib_stats: Processed 32 operations (0.00us average, 0% utilization) in the last 10min
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 108: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=108, ref=pe_calc-dc-1269428002-131, seq=848, quorate=0
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 16: 0 actions in 0 synapses
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 16 (ref=pe_calc-dc-1269428002-131) derived from /var/lib/pengine/pe-input-338.bz2
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:53:22 ha-test1 crmd: [9046]: notice: run_graph: Transition 16 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-338.bz2): Complete
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 16 is now complete
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: notify_crmd: Transition 16 status: done - <null>
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: process_pe_message: Transition 16: PEngine Input stored in: /var/lib/pengine/pe-input-338.bz2
Mar 24 11:56:12 ha-test1 cib: [9042]: info: cib_stats: Processed 33 operations (606.00us average, 0% utilization) in the last 10min
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="132" num_updates="52" >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="133" num_updates="1" >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 109: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.133.1): ok (rc=0)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=109, ref=pe_calc-dc-1269428177-132, seq=848, quorate=0
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:56:17 ha-test1 cib: [22665]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-71.raw
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Demote resDRBD:0#011(Master -> Stopped ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDRBD:0#011(ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDLM:1#011(ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resO2CB:1#011(ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resFS:1#011(ha-test1)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 17: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-465.bz2
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 17: 31 actions in 31 synapses
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 17 (ref=pe_calc-dc-1269428177-132) derived from /var/lib/pengine/pe-warn-465.bz2
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 39 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 62 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 84: notify resDRBD:0_pre_notify_demote_0 on ha-test1 (local)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=84:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:45: notify
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 86: notify resDRBD:1_pre_notify_demote_0 on ha-test2
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: stop resFS:1_stop_0 on ha-test1 (local)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=59:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_stop_0 )
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: rsc:resFS:1:46: stop
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=40, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:17 ha-test1 cib: [22665]: info: write_cib_contents: Wrote version 0.133.0 of the CIB to disk (digest: 4ac14c3b34ff566124e9c8cc8bd23cff)
Mar 24 11:56:17 ha-test1 cib: [22665]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.RpSDV6 (digest: /var/lib/heartbeat/crm/cib.4ylhzK)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_demote_0 (86) confirmed on ha-test2 (rc=0)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=45, rc=0, cib-update=110, confirmed=true) ok
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_demote_0 (84) confirmed on ha-test1 (rc=0)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 40 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: demote resDRBD:0_demote_0 on ha-test1 (local)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_demote_0 )
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:47: demote
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=35, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:17 ha-test1 Filesystem[22667]: INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:56:17 INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:56:17 ha-test1 Filesystem[22667]: INFO: Trying to unmount /nfs
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:56:17 INFO: Trying to unmount /nfs
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0: State change failed: (-12) Device is held open by someone
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) Command '
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) drbdsetup
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) secondary
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) ' terminated with exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0: State change failed: (-12) Device is held open by someone
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) Command '
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) drbdsetup
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) secondary
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) ' terminated with exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test2 crm_shadow: [12243]: info: Invoked: crm_shadow 
Mar 24 11:56:17 ha-test2 crm_attribute: [12248]: info: Invoked: crm_attribute -N ha-test1 -n standby -v on -l forever 
Mar 24 11:56:17 ha-test2 cib: [12249]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-51.raw
Mar 24 11:56:17 ha-test2 cib: [12249]: info: write_cib_contents: Wrote version 0.133.0 of the CIB to disk (digest: 68d6db75d4871405c3ab119cbd61ae10)
Mar 24 11:56:17 ha-test2 cib: [12249]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.rwPUa0 (digest: /var/lib/heartbeat/crm/cib.yIn479)
Mar 24 11:56:17 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=86:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:17 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:19: notify
Mar 24 11:56:17 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=19, rc=0, cib-update=26, confirmed=true) ok
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 Filesystem[22667]: INFO: unmounted /nfs successfully
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:56:18 INFO: unmounted /nfs successfully
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Command output: 
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_stop_0 (call=46, rc=0, cib-update=111, confirmed=true) ok
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_stop_0 (59) confirmed on ha-test1 (rc=0)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: stop resO2CB:1_stop_0 on ha-test1 (local)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=52:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_stop_0 )
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:48: stop
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=38, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:56:18 ha-test1 o2cb[22818]: INFO: Stopping resO2CB:1
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:56:18 INFO: Stopping resO2CB:1
Mar 24 11:56:18 ha-test1 o2cb[22818]: INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:56:18 INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) /usr/lib/ocf/resource.d//pacemaker/o2cb: line 169: killproc: command not found
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_stop_0 (call=48, rc=0, cib-update=112, confirmed=true) ok
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_stop_0 (52) confirmed on ha-test1 (rc=0)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: stop resDLM:1_stop_0 on ha-test1 (local)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_stop_0 )
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:49: stop
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=36, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:18 ha-test1 ocfs2_controld[15089]: Error from dlmc_fs_result: Bad file descriptor
Mar 24 11:56:18 ha-test1 ocfs2_controld[15089]: dlmcontrol connection died
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:stop:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:56:19 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:56:19 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_stop_0 (call=49, rc=0, cib-update=113, confirmed=true) ok
Mar 24 11:56:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_stop_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:56:19 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 49 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_demote_0 (call=47, rc=0, cib-update=114, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_demote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 38 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 41 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 85: notify resDRBD:0_post_notify_demote_0 on ha-test1 (local)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=85:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:50: notify
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 87: notify resDRBD:1_post_notify_demote_0 on ha-test2
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=50, rc=0, cib-update=115, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_demote_0 (85) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_demote_0 (87) confirmed on ha-test2 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 42 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 79: notify resDRBD:0_pre_notify_stop_0 on ha-test1 (local)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=79:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:51: notify
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 80: notify resDRBD:1_pre_notify_stop_0 on ha-test2
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=51, rc=0, cib-update=116, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_stop_0 (79) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_stop_0 (80) confirmed on ha-test2 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 25 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 14: stop resDRBD:0_stop_0 on ha-test1 (local)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=14:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_stop_0 )
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:52: stop
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (-INFINITY)
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 52: master-resDRBD:0=-INFINITY
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.133.12) : Transient attribute: update
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 crm_attribute: [22976]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent delete 54: node=ha-test1, attr=master-resDRBD:0, id=<n/a>, set=(null), section=status
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.133.13) : Transient attribute: removal
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_stop_0 (call=52, rc=0, cib-update=117, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_stop_0 (14) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:1_post_notify_stop_0 on ha-test2
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_stop_0 (81) confirmed on ha-test2 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 30 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:20 ha-test1 crmd: [9046]: notice: run_graph: Transition 17 (Complete=30, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-warn-465.bz2): Stopped
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 17 is now complete
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 118: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=118, ref=pe_calc-dc-1269428180-145, seq=848, quorate=0
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Stopped)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 18: 0 actions in 0 synapses
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 18 (ref=pe_calc-dc-1269428180-145) derived from /var/lib/pengine/pe-warn-466.bz2
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:20 ha-test1 crmd: [9046]: notice: run_graph: Transition 18 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-466.bz2): Complete
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 18 is now complete
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: notify_crmd: Transition 18 status: done - <null>
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 18: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-466.bz2
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=87:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:20: notify
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=20, rc=0, cib-update=27, confirmed=true) ok
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=80:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:21: notify
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=21, rc=0, cib-update=28, confirmed=true) ok
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=81:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:22: notify
Mar 24 11:56:20 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:56:20 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=22, rc=0, cib-update=29, confirmed=true) ok
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="133" num_updates="15" >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="134" num_updates="1" >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 119: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.134.1): ok (rc=0)
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=119, ref=pe_calc-dc-1269428185-146, seq=848, quorate=0
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:25 ha-test1 cib: [22977]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-72.raw
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:0#011(ha-test1)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 19: 11 actions in 11 synapses
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 19 (ref=pe_calc-dc-1269428185-146) derived from /var/lib/pengine/pe-warn-467.bz2
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 70: notify resDRBD:1_pre_notify_start_0 on ha-test2
Mar 24 11:56:25 ha-test1 cib: [22977]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: fe287a4e655a0b728685dfff20428f47)
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 19: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-467.bz2
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:56:25 ha-test1 cib: [22977]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.ad2qpz (digest: /var/lib/heartbeat/crm/cib.lcORwF)
Mar 24 11:56:25 ha-test2 crm_shadow: [12404]: info: Invoked: crm_shadow 
Mar 24 11:56:25 ha-test2 crm_attribute: [12408]: info: Invoked: crm_attribute -N ha-test1 -n standby -v off -l forever 
Mar 24 11:56:25 ha-test2 cib: [12409]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-52.raw
Mar 24 11:56:25 ha-test2 cib: [12409]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: 25c404d549614fcfa017a9957e21fa94)
Mar 24 11:56:25 ha-test2 cib: [12409]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.EggLDs (digest: /var/lib/heartbeat/crm/cib.XXgL34)
Mar 24 11:56:25 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=70:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:25 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:23: notify
Mar 24 11:56:25 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:56:25 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=23, rc=0, cib-update=30, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_start_0 (70) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: start resDRBD:0_start_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=9:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_start_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:53: start
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout)         allow-two-primaries;
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:56:26 ha-test1 crm_attribute: [23030]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_start_0 (call=53, rc=0, cib-update=120, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_start_0 (9) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 69: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=69:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:54: notify
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 71: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=54, rc=0, cib-update=121, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (69) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (71) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=10:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_30000 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:55: monitor
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10)
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 58: master-resDRBD:0=10
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.134.6) : Transient attribute: update
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=55, rc=0, cib-update=122, confirmed=false) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (10) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:26 ha-test1 crmd: [9046]: notice: run_graph: Transition 19 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-467.bz2): Complete
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 19 is now complete
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 123: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=123, ref=pe_calc-dc-1269428186-152, seq=848, quorate=0
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:0#011(Slave -> Master ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:1#011(ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 20: 25 actions in 25 synapses
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 20 (ref=pe_calc-dc-1269428186-152) derived from /var/lib/pengine/pe-input-339.bz2
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 1: cancel resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: send_direct_ack: ACK'ing resource op resDRBD:0_monitor_30000 from 1:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269428186-154
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269428186-154 from ha-test1
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (1) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=55, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=81:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:56: notify
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 83: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: start resDLM:1_start_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_start_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:57: start
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: process_pe_message: Transition 20: PEngine Input stored in: /var/lib/pengine/pe-input-339.bz2
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (83) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 23126 (0x88624c0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=56, rc=0, cib-update=125, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (81) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: promote resDRBD:0_promote_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_promote_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:58: promote
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout)         allow-two-primaries;
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout) 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_promote_0 (call=58, rc=0, cib-update=126, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_promote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 82: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=82:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:59: notify
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 84: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (84) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 60: master-resDRBD:0=10000
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.134.13) : Transient attribute: update
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=59, rc=0, cib-update=127, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (82) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=71:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:24: notify
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=24, rc=0, cib-update=31, confirmed=true) ok
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=83:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:25: notify
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=25, rc=0, cib-update=32, confirmed=true) ok
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: do_lrm_rsc_op: Performing key=84:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: rsc:resDRBD:1:26: notify
Mar 24 11:56:26 ha-test2 lrmd: [1489]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:56:26 ha-test2 crmd: [1492]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=26, rc=0, cib-update=33, confirmed=true) ok
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_start_0 (call=57, rc=0, cib-update=128, confirmed=true) ok
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_start_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:27 ha-test1 crmd: [9046]: notice: run_graph: Transition 20 (Complete=17, Pending=0, Fired=0, Skipped=6, Incomplete=2, Source=/var/lib/pengine/pe-input-339.bz2): Stopped
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 20 is now complete
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 129: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=129, ref=pe_calc-dc-1269428187-161, seq=848, quorate=0
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:27 ha-test1 crmd: [9046]: WARN: destroy_action: Cancelling timer for action 1 (src=246)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 21: 10 actions in 10 synapses
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 21 (ref=pe_calc-dc-1269428187-161) derived from /var/lib/pengine/pe-input-340.bz2
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: monitor resDRBD:0_monitor_20000 on ha-test1 (local)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:21:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_20000 )
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:60: monitor
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 46: monitor resDLM:1_monitor_120000 on ha-test1 (local)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=46:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_monitor_120000 )
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:61: monitor
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 53: start resO2CB:1_start_0 on ha-test1 (local)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=53:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_start_0 )
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:62: start
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: process_pe_message: Transition 21: PEngine Input stored in: /var/lib/pengine/pe-input-340.bz2
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=61, rc=0, cib-update=130, confirmed=false) ok
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_monitor_120000 (46) confirmed on ha-test1 (rc=0)
Mar 24 11:56:27 ha-test1 o2cb[23213]: INFO: ocfs2 not loaded
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:56:27 INFO: ocfs2 not loaded
Mar 24 11:56:27 ha-test1 o2cb[23213]: INFO: Starting resO2CB:1
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:56:27 INFO: Starting resO2CB:1
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=60, rc=8, cib-update=131, confirmed=false) master
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_20000 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:56:27 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 23285 (0x8859930)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_start_0 (call=62, rc=0, cib-update=132, confirmed=true) ok
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_start_0 (53) confirmed on ha-test1 (rc=0)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 54: monitor resO2CB:1_monitor_120000 on ha-test1 (local)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=54:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_monitor_120000 )
Mar 24 11:56:29 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:63: monitor
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 61: start resFS:1_start_0 on ha-test1 (local)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=61:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_start_0 )
Mar 24 11:56:29 ha-test1 lrmd: [9043]: info: rsc:resFS:1:64: start
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=63, rc=0, cib-update=133, confirmed=false) ok
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_monitor_120000 (54) confirmed on ha-test1 (rc=0)
Mar 24 11:56:30 ha-test1 Filesystem[23298]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:56:30 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:start:stderr) 2010/03/24_11:56:30 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_start_0 (call=64, rc=0, cib-update=134, confirmed=true) ok
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_start_0 (61) confirmed on ha-test1 (rc=0)
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 62: monitor resFS:1_monitor_120000 on ha-test1 (local)
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=62:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_monitor_120000 )
Mar 24 11:56:30 ha-test1 lrmd: [9043]: info: rsc:resFS:1:65: monitor
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=65, rc=0, cib-update=135, confirmed=false) ok
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_monitor_120000 (62) confirmed on ha-test1 (rc=0)
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:30 ha-test1 crmd: [9046]: notice: run_graph: Transition 21 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-340.bz2): Complete
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 21 is now complete
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: notify_crmd: Transition 21 status: done - <null>
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:56:52 ha-test1 corosync[9029]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:56:52 ha-test2 corosync[1466]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:56:52 ha-test2 corosync[1466]:   [TOTEM ] The network interface is down.
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 852: memb=1, new=0, lost=1
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: lost: ha-test2 1550025994
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 852: memb=1, new=0, lost=0
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: ais_mark_unseen_peer_dead: Node ha-test2 was not seen in the previous transition
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: lost
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 852 to 0 children
Mar 24 11:56:56 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:56:56 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:56:56 ha-test1 cib: [9042]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:56:56 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now lost (was member)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: erase_node_from_join: Removed node ha-test2 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
Mar 24 11:56:56 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/136, version=0.134.21): ok (rc=0)
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="852" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="848" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:56:56 ha-test1 crmd: [9046]: WARN: match_down_event: No match for shutdown action on ha-test2
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: te_update_diff: Stonith/shutdown of ha-test2 not matched
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=ha-test2, magic=NA, cib=0.134.22) : Node failure
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 138: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=138, ref=pe_calc-dc-1269428216-169, seq=852, quorate=0
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:1 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 22: 0 actions in 0 synapses
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 22 (ref=pe_calc-dc-1269428216-169) derived from /var/lib/pengine/pe-warn-468.bz2
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:56 ha-test1 crmd: [9046]: notice: run_graph: Transition 22 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-468.bz2): Complete
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 22 is now complete
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: notify_crmd: Transition 22 status: done - <null>
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 22: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-468.bz2
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 856: memb=1, new=0, lost=0
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 856: memb=2, new=1, lost=0
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: member
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: NEW:  ha-test2 1550025994
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:57:18 ha-test1 cib: [9042]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 cib: [9042]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: 0x88590d0 Node 1550025994 (ha-test2) born on: 856
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now member (was lost)
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/139, version=0.134.23): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=local/crmd/140, version=0.134.24): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/141, version=0.134.24): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=ha-test2/ha-test2/(null), version=0.134.25): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=resDLM:0_monitor_0, magic=0:7;10:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011, cib=0.134.23) : Resource op removal
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.134.24) : Transient attribute: removal
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/143, version=0.134.25): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="856" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_state_transition: Membership changed: 848 -> 856 - join restart
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 145: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: join_make_offer: Making join offers based on membership 856
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [TOTEM ] The network interface [10.133.99.92] is now up.
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: process_ais_conf: Reading configure
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_init: Local handle: 2730409743423111170 for logging
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_next: Processing additional logging options...
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Found 'off' for option: debug
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'off' for option: to_file
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Found 'daemon' for option: syslog_facility
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_init: Local handle: 5880381755227111427 for service
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: config_find_next: Processing additional service options...
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: CRM: Initialized
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] Logging: Initialized pcmk_startup
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: Maximum core file size is: 4294967295
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: Service: 9
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_startup: Local hostname: ha-test2
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_update_nodeid: Local node id: 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Creating entry for node 1550025994 born on 0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96b9f98 Node 1550025994 now known as ha-test2 (was: (null))
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test2 now has 1 quorum votes (was 0)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: member
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1493 for process stonithd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1494 for process cib
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1495 for process lrmd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1496 for process attrd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1497 for process pengine
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: spawn_child: Forked child 1498 for process crmd
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: Pacemaker Cluster Manager 1.0.7
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Mar 24 11:57:18 ha-test2 lrmd: [1495]: info: Started.
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: Stack hogger failed 0xffffffff
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: Invoked: /usr/lib/heartbeat/attrd 
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Starting up
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 cib: [1494]: info: Invoked: /usr/lib/heartbeat/cib 
Mar 24 11:57:18 ha-test2 cib: [1494]: info: G_main_add_TriggerHandler: Added signal manual handler
Mar 24 11:57:18 ha-test2 cib: [1494]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 cib: [1494]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Mar 24 11:57:18 ha-test2 pengine: [1497]: info: Invoked: /usr/lib/heartbeat/pengine 
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Mar 24 11:57:18 ha-test2 pengine: [1497]: info: main: Starting pengine
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync configuration service
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Mar 24 11:57:18 ha-test2 cib: [1494]: info: startCib: CIB Initialization completed successfully
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 cib: [1494]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync profile loading service
Mar 24 11:57:18 ha-test2 corosync[1460]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 852: memb=0, new=0, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 852: memb=1, new=1, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: NEW:  ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test2 now has process list: 00000000000000000000000000013312 (78610)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:57:18 ha-test2 corosync[1460]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 cib: [1494]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c0350 for attrd/1496
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c0658 for cib/1494
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 cib: [1494]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Sending membership update 852 to (null)
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Cluster connection active
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_init: Starting cib mainloop
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Accepting attribute updates
Mar 24 11:57:18 ha-test2 cib: [1494]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:57:18 ha-test2 attrd: [1496]: info: main: Starting mainloop...
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  (new) votes=1 (new) born=0 seen=0 proc=00000000000000000000000000000354(null)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: Invoked: /usr/lib/heartbeat/crmd 
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: main: CRM Hg Version: 54d7869bfe3691eb723b1d47810e5585d8246b58
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crmd_init: Starting crmd
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_cib_control: CIB connection established
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c2560 for crmd/1498
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Sending membership update 852 to (null)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_ha_control: Connected to the cluster
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_started: Delaying start, CCM (0000000000100000) not connected
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crmd_init: Starting crmd's mainloop
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  (new) votes=1 (new) born=0 seen=0 proc=00000000000000000000000000000354(null)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_started: Delaying start, Config not read (0000000000000040)
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: config_query_callback: Checking for expired actions every 900000ms
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_expected_votes: Expected quorum votes 1024 -> 2
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: crm_cluster_connect: Connecting to OpenAIS
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: init_ais_connection: Creating connection to our AIS plugin
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_started: The local CRM is operational
Mar 24 11:57:18 ha-test2 crmd: [1498]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: init_ais_connection: AIS connection established
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x96c2cf0 for stonithd/1493
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: get_ais_nodeid: Server details: id=1550025994 uname=ha-test2 cname=pcmk
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: crm_new_peer: Node ha-test2 now has id: 1550025994
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: crm_new_peer: Node 1550025994 is now known as ha-test2
Mar 24 11:57:18 ha-test2 stonithd: [1493]: notice: /usr/lib/heartbeat/stonithd start up successfully.
Mar 24 11:57:18 ha-test2 stonithd: [1493]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Mar 24 11:57:18 ha-test2 cib: [1501]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-53.raw
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 856: memb=1, new=0, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 856: memb=2, new=1, lost=0
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Creating entry for node 1533248778 born on 856
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node 1533248778/unknown is now: member
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: NEW:  .pending. 1533248778
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: MEMB: .pending. 1533248778
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96b9f98 Node 1550025994 ((null)) born on: 856
Mar 24 11:57:18 ha-test2 corosync[1460]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:57:18 ha-test2 cib: [1494]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_new_peer: Node <null> now has id: 1533248778
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_update_peer: Node (null): id=1533248778 state=member (new) addr=r(0) ip(10.133.99.91)  votes=0 born=0 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96c2a30 Node 1533248778 (ha-test1) born on: 840
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: 0x96c2a30 Node 1533248778 now known as ha-test1 (was: (null))
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test1 now has process list: 00000000000000000000000000013312 (78610)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: update_member: Node ha-test1 now has 1 quorum votes (was 0)
Mar 24 11:57:18 ha-test2 corosync[1460]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test2 cib: [1494]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_get_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:57:18 ha-test2 cib: [1494]: info: crm_update_peer: Node ha-test1: id=1533248778 state=member addr=r(0) ip(10.133.99.91)  votes=1 (new) born=840 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_process_diff: Diff 0.134.22 -> 0.134.23 not applied to 0.134.0: current "num_updates" is less than required
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_server_process_diff: Requesting re-sync from peer
Mar 24 11:57:18 ha-test2 corosync[1460]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:57:18 ha-test2 cib: [1494]: WARN: cib_diff_notify: Local-only Change (client:crmd, call: 139): 0.0.0 (Application of an update diff failed, requesting a full refresh)
Mar 24 11:57:18 ha-test2 cib: [1501]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: cc1d42d0a4fb39a8b5cc16b326f29393)
Mar 24 11:57:18 ha-test2 cib: [1494]: WARN: cib_server_process_diff: Not applying diff 0.134.23 -> 0.134.24 (sync in progress)
Mar 24 11:57:18 ha-test2 cib: [1501]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.NDF4ks (digest: /var/lib/heartbeat/crm/cib.iKqto0)
Mar 24 11:57:18 ha-test2 cib: [1494]: WARN: cib_server_process_diff: Not applying diff 0.134.24 -> 0.134.25 (sync in progress)
Mar 24 11:57:18 ha-test2 cib: [1494]: info: cib_replace_notify: Replaced: 0.134.0 -> 0.134.25 from ha-test1
Mar 24 11:57:18 ha-test2 cib: [1513]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-54.raw
Mar 24 11:57:18 ha-test2 cib: [1513]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: 100caec81dbb156797a7fce9abd1d580)
Mar 24 11:57:18 ha-test2 cib: [1513]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.taBEFs (digest: /var/lib/heartbeat/crm/cib.jTdD30)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: A new node joined the cluster
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_finalize: join-5: Syncing the CIB from ha-test1 to the rest of the cluster
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/148, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/149, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/150, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=ha-test2/crmd/7, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=ha-test2/crmd/8, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-5: Updating node state to member for ha-test2
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/151, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-5: Updating node state to member for ha-test1
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test1']/lrm (origin=local/crmd/153, version=0.134.27): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test1']/lrm": ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/155, version=0.134.28): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: crm_update_quorum: Updating quorum status to false (call=157)
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/157, version=0.134.28): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 158: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=158, ref=pe_calc-dc-1269428239-181, seq=856, quorate=0
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (<null>)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:1#011(ha-test2)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 23: 17 actions in 17 synapses
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 23 (ref=pe_calc-dc-1269428239-181) derived from /var/lib/pengine/pe-warn-469.bz2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: monitor resDRBD:1_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDLM:0_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 11: monitor resO2CB:0_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 12: monitor resFS:0_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 77: notify resDRBD:0_pre_notify_start_0 on ha-test1 (local)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=77:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:19 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:66: notify
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 23: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-469.bz2
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:57:19 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=66, rc=0, cib-update=159, confirmed=true) ok
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_start_0 (77) confirmed on ha-test1 (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_0 (11) confirmed on ha-test2 (rc=0)
Mar 24 11:57:19 ha-test2 logd: [1854]: info: logd started with default configuration.
Mar 24 11:57:19 ha-test2 logd: [1854]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:57:19 ha-test2 logd: [1857]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_new_peer: Node <null> now has id: 1533248778
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_update_peer: Node (null): id=1533248778 state=member (new) addr=r(0) ip(10.133.99.91)  votes=0 born=0 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_get_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: crm_update_peer: Node ha-test1: id=1533248778 state=member addr=r(0) ip(10.133.99.91)  votes=1 (new) born=840 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: update_attrd: Connecting to attrd...
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for terminate
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for shutdown
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: crm_new_peer: Node ha-test1 now has id: 1533248778
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: crm_new_peer: Node 1533248778 is now known as ha-test1
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation shutdown=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for master-resDRBD:0
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:0=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for probe_complete
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation probe_complete=<null>: cib not connected
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: find_hash_entry: Creating hash entry for master-resDRBD:1
Mar 24 11:57:19 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:1=<null>: cib not connected
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=9:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:2: monitor
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=10:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resDLM:0:3: monitor
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=11:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resO2CB:0:4: monitor
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=12:23:7:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_0 )
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: rsc:resFS:0:5: monitor
Mar 24 11:57:19 ha-test2 o2cb[1950]: INFO: configfs not laoded
Mar 24 11:57:19 ha-test2 Filesystem[1951]: WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: RA output: (resO2CB:0:monitor:stderr) 2010/03/24_11:57:19 INFO: configfs not laoded
Mar 24 11:57:19 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resO2CB:0_monitor_0 (call=4, rc=7, cib-update=9, confirmed=true) not running
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: RA output: (resFS:0:monitor:stderr) 2010/03/24_11:57:19 WARNING: Couldn't find device [/dev/drbd0]. Expected /dev/??? to exist
Mar 24 11:57:19 ha-test2 lrmd: [1495]: info: RA output: (resDLM:0:monitor:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_0 (10) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_0 (9) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_0 (12) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on ha-test2 - no waiting
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 6 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: start resDRBD:1_start_0 on ha-test2
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_start_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 78: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=78:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:67: notify
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 79: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDLM:0_monitor_0 (call=3, rc=7, cib-update=10, confirmed=true) not running
Mar 24 11:57:20 ha-test2 crm_attribute: [2062]: info: Invoked: crm_attribute -N ha-test2 -n master-resDRBD:1 -l reboot -D 
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_0 (call=2, rc=7, cib-update=11, confirmed=true) not running
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resFS:0_monitor_0 (call=5, rc=7, cib-update=12, confirmed=true) not running
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation probe_complete=true: cib not connected
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=17:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_start_0 )
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:6: start
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout)         allow-two-primaries;
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:57:20 ha-test2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/2.6.33.1/kernel/drivers/crypto/padlock-sha.ko): No such device
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:57:20 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:1=10000: cib not connected
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:start:stdout) 
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_start_0 (call=6, rc=0, cib-update=13, confirmed=true) ok
Mar 24 11:57:20 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=79:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:57:20 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:7: notify
Mar 24 11:57:21 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=67, rc=0, cib-update=160, confirmed=true) ok
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (78) confirmed on ha-test1 (rc=0)
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (79) confirmed on ha-test2 (rc=0)
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:57:21 ha-test1 crmd: [9046]: notice: run_graph: Transition 23 (Complete=17, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-469.bz2): Complete
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 23 is now complete
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: notify_crmd: Transition 23 status: done - <null>
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:57:21 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:57:21 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=7, rc=0, cib-update=14, confirmed=true) ok
Mar 24 11:57:21 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=18:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_30000 )
Mar 24 11:57:21 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:57:21 ha-test2 attrd: [1496]: info: attrd_perform_update: Delaying operation master-resDRBD:1=10000: cib not connected
Mar 24 11:57:21 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, rc=0, cib-update=15, confirmed=false) ok
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.134.38) : Transient attribute: update
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 161: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.134.39) : Transient attribute: update
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 162: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=162, ref=pe_calc-dc-1269428243-192, seq=856, quorate=0
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test2 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:1#011(Slave -> Master ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:0#011(ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:0#011(ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:0#011(ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 24: 25 actions in 25 synapses
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 24 (ref=pe_calc-dc-1269428243-192) derived from /var/lib/pengine/pe-input-341.bz2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 5: cancel resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 85: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=85:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:23 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:68: notify
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 87: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 43: start resDLM:0_start_0 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269428243-6 from ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (5) confirmed on ha-test2 (rc=0)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: process_pe_message: Transition 24: PEngine Input stored in: /var/lib/pengine/pe-input-341.bz2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=68, rc=0, cib-update=163, confirmed=true) ok
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (85) confirmed on ha-test1 (rc=0)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (87) confirmed on ha-test2 (rc=0)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: promote resDRBD:1_promote_0 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_promote_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 86: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=86:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:23 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:69: notify
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 88: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:57:23 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=69, rc=0, cib-update=164, confirmed=true) ok
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (86) confirmed on ha-test1 (rc=0)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: cib_connect: Connected to the CIB after 1 signon attempts
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: cib_connect: Sending full refresh
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_perform_update: Sent update 4: probe_complete=true
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (10000)
Mar 24 11:57:23 ha-test2 attrd: [1496]: info: attrd_perform_update: Sent update 10: master-resDRBD:1=10000
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: send_direct_ack: ACK'ing resource op resDRBD:1_monitor_30000 from 5:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269428243-6
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_30000 (call=8, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=87:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:9: notify
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=43:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_start_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDLM:0:10: start
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: RA output: (resDLM:0:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=9, rc=0, cib-update=17, confirmed=true) ok
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=17:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_promote_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:11: promote
Mar 24 11:57:23 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 3372 (0x96c3628)
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:promote:stdout)         allow-two-primaries;
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:promote:stdout) 
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_promote_0 (call=11, rc=0, cib-update=18, confirmed=true) ok
Mar 24 11:57:23 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=88:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_notify_0 )
Mar 24 11:57:23 ha-test2 lrmd: [1495]: info: rsc:resDRBD:1:12: notify
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (88) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_20000 on ha-test2
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_20000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_start_0 (43) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 44: monitor resDLM:0_monitor_120000 on ha-test2
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 51: start resO2CB:0_start_0 on ha-test2
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_120000 (44) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: RA output: (resDRBD:1:notify:stdout) 
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_notify_0 (call=12, rc=0, cib-update=19, confirmed=true) ok
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=18:24:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:1_monitor_20000 )
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDRBD:1_monitor_20000 (call=13, rc=8, cib-update=20, confirmed=false) master
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDLM:0_start_0 (call=10, rc=0, cib-update=21, confirmed=true) ok
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=44:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:0_monitor_120000 )
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=51:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_start_0 )
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: rsc:resO2CB:0:15: start
Mar 24 11:57:24 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resDLM:0_monitor_120000 (call=14, rc=0, cib-update=22, confirmed=false) ok
Mar 24 11:57:24 ha-test2 o2cb[3704]: INFO: Stack glue driver not loaded
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:57:24 INFO: Stack glue driver not loaded
Mar 24 11:57:24 ha-test2 o2cb[3704]: INFO: Starting resO2CB:0
Mar 24 11:57:24 ha-test2 lrmd: [1495]: info: RA output: (resO2CB:0:start:stderr) 2010/03/24_11:57:24 INFO: Starting resO2CB:0
Mar 24 11:57:25 ha-test2 corosync[1460]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 3758 (0x96c8648)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_start_0 (51) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: monitor resO2CB:0_monitor_120000 on ha-test2
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: start resFS:0_start_0 on ha-test2
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_120000 (52) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_start_0 (59) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 60: monitor resFS:0_monitor_120000 on ha-test2
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_120000 (60) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:57:27 ha-test1 crmd: [9046]: notice: run_graph: Transition 24 (Complete=25, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-341.bz2): Complete
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 24 is now complete
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: notify_crmd: Transition 24 status: done - <null>
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resO2CB:0_start_0 (call=15, rc=0, cib-update=23, confirmed=true) ok
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=52:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:0_monitor_120000 )
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=59:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_start_0 )
Mar 24 11:57:27 ha-test2 lrmd: [1495]: info: rsc:resFS:0:17: start
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resO2CB:0_monitor_120000 (call=16, rc=0, cib-update=24, confirmed=false) ok
Mar 24 11:57:27 ha-test2 Filesystem[3979]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:57:27 ha-test2 lrmd: [1495]: info: RA output: (resFS:0:start:stderr) 2010/03/24_11:57:27 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resFS:0_start_0 (call=17, rc=0, cib-update=25, confirmed=true) ok
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: do_lrm_rsc_op: Performing key=60:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:0_monitor_120000 )
Mar 24 11:57:27 ha-test2 crmd: [1498]: info: process_lrm_event: LRM operation resFS:0_monitor_120000 (call=18, rc=0, cib-update=26, confirmed=false) ok
Mar 24 11:59:02 ha-test2 cibadmin: [4292]: info: Invoked: cibadmin -Ql -o nodes 
Mar 24 11:59:03 ha-test1 logger: Mark:HB_REPORT:1269428341
Mar 24 11:59:03 ha-test2 root: Mark:HB_REPORT:1269428341
