Mar 24 11:02:21 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 48: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=48, ref=pe_calc-dc-1269424941-53, seq=840, quorate=0
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:02:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 4: 0 actions in 0 synapses
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1269424941-53) derived from /var/lib/pengine/pe-input-332.bz2
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:02:21 ha-test1 crmd: [9046]: notice: run_graph: Transition 4 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-332.bz2): Complete
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 4 is now complete
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: notify_crmd: Transition 4 status: done - <null>
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:02:21 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:02:21 ha-test1 pengine: [9045]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/pengine/pe-input-332.bz2
Mar 24 11:06:12 ha-test1 cib: [9042]: info: cib_stats: Processed 33 operations (606.00us average, 0% utilization) in the last 10min
Mar 24 11:16:12 ha-test1 cib: [9042]: info: cib_stats: Processed 32 operations (312.00us average, 0% utilization) in the last 10min
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 49: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=49, ref=pe_calc-dc-1269425841-54, seq=840, quorate=0
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:17:21 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 5: 0 actions in 0 synapses
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 5 (ref=pe_calc-dc-1269425841-54) derived from /var/lib/pengine/pe-input-333.bz2
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:17:21 ha-test1 crmd: [9046]: notice: run_graph: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-333.bz2): Complete
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 5 is now complete
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: notify_crmd: Transition 5 status: done - <null>
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:17:21 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:17:21 ha-test1 pengine: [9045]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-333.bz2
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="130" num_updates="42" >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="131" num_updates="1" >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 50: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:19:28 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.131.1): ok (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=50, ref=pe_calc-dc-1269425968-55, seq=840, quorate=0
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Demote resDRBD:0#011(Master -> Stopped ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDRBD:0#011(ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDLM:1#011(ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resO2CB:1#011(ha-test1)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:28 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resFS:1#011(ha-test1)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:28 ha-test1 cib: [14490]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-69.raw
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 6: 31 actions in 31 synapses
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1269425968-55) derived from /var/lib/pengine/pe-warn-459.bz2
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 39 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 62 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 80: notify resDRBD:0_pre_notify_demote_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=80:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:20: notify
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 82: notify resDRBD:1_pre_notify_demote_0 on ha-test2
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: stop resFS:1_stop_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=59:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_stop_0 )
Mar 24 11:19:28 ha-test1 cib: [14490]: info: write_cib_contents: Wrote version 0.131.0 of the CIB to disk (digest: 0ba795530b6b6d0f945897bcb70a2107)
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resFS:1:21: stop
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=19, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:28 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 6: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-459.bz2
Mar 24 11:19:28 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:19:28 ha-test1 cib: [14490]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.9jHvtg (digest: /var/lib/heartbeat/crm/cib.Bnb1E3)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_demote_0 (82) confirmed on ha-test2 (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=20, rc=0, cib-update=51, confirmed=true) ok
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_demote_0 (80) confirmed on ha-test1 (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 40 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: demote resDRBD:0_demote_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_demote_0 )
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:22: demote
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=14, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:28 ha-test1 Filesystem[14492]: INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:19:28 INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:19:28 ha-test1 Filesystem[14492]: INFO: Trying to unmount /nfs
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:19:28 INFO: Trying to unmount /nfs
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0: State change failed: (-12) Device is held open by someone
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) Command '
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) drbdsetup
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) secondary
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) ' terminated with exit code 11
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Exit code 11
Mar 24 11:19:28 ha-test1 drbd[14536]: ERROR: r0: Command output: 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:19:28 ERROR: r0: Command output: 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:19:28 ha-test1 Filesystem[14492]: INFO: unmounted /nfs successfully
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:19:28 INFO: unmounted /nfs successfully
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) /dev/drbd0: Wrong medium type
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_stop_0 (call=21, rc=0, cib-update=52, confirmed=true) ok
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_stop_0 (59) confirmed on ha-test1 (rc=0)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: stop resO2CB:1_stop_0 on ha-test1 (local)
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=52:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_stop_0 )
Mar 24 11:19:28 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:23: stop
Mar 24 11:19:28 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=17, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:29 ha-test1 o2cb[14635]: INFO: Stopping resO2CB:1
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:19:29 INFO: Stopping resO2CB:1
Mar 24 11:19:29 ha-test1 o2cb[14635]: INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:19:29 INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) /usr/lib/ocf/resource.d//pacemaker/o2cb: line 169: killproc: command not found
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_stop_0 (call=23, rc=0, cib-update=53, confirmed=true) ok
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_stop_0 (52) confirmed on ha-test1 (rc=0)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: stop resDLM:1_stop_0 on ha-test1 (local)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_stop_0 )
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:24: stop
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=15, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:29 ha-test1 ocfs2_controld[9480]: Error from dlmc_fs_result: Bad file descriptor
Mar 24 11:19:29 ha-test1 ocfs2_controld[9480]: dlmcontrol connection died
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:stop:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_demote_0 (call=22, rc=0, cib-update=54, confirmed=true) ok
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_demote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 38 fired and confirmed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 41 fired and confirmed
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:0_post_notify_demote_0 on ha-test1 (local)
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=81:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:29 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:25: notify
Mar 24 11:19:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 83: notify resDRBD:1_post_notify_demote_0 on ha-test2
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=25, rc=0, cib-update=55, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_demote_0 (81) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_demote_0 (83) confirmed on ha-test2 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 42 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 75: notify resDRBD:0_pre_notify_stop_0 on ha-test1 (local)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=75:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:26: notify
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 76: notify resDRBD:1_pre_notify_stop_0 on ha-test2
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=26, rc=0, cib-update=56, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_stop_0 (75) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_stop_0 (call=24, rc=0, cib-update=57, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_stop_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 49 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_stop_0 (76) confirmed on ha-test2 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 25 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 14: stop resDRBD:0_stop_0 on ha-test1 (local)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=14:6:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_stop_0 )
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:27: stop
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (-INFINITY)
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 19: master-resDRBD:0=-INFINITY
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.131.12) : Transient attribute: update
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 crm_attribute: [14780]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:19:30 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent delete 21: node=ha-test1, attr=master-resDRBD:0, id=<n/a>, set=(null), section=status
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.131.13) : Transient attribute: removal
Mar 24 11:19:30 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_stop_0 (call=27, rc=0, cib-update=58, confirmed=true) ok
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_stop_0 (14) confirmed on ha-test1 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 77: notify resDRBD:1_post_notify_stop_0 on ha-test2
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_stop_0 (77) confirmed on ha-test2 (rc=0)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 30 fired and confirmed
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:30 ha-test1 crmd: [9046]: notice: run_graph: Transition 6 (Complete=30, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-warn-459.bz2): Stopped
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 6 is now complete
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 59: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=59, ref=pe_calc-dc-1269425970-68, seq=840, quorate=0
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Stopped)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:30 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 7: 0 actions in 0 synapses
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1269425970-68) derived from /var/lib/pengine/pe-warn-460.bz2
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:30 ha-test1 crmd: [9046]: notice: run_graph: Transition 7 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-460.bz2): Complete
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 7 is now complete
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: notify_crmd: Transition 7 status: done - <null>
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:30 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:19:30 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 7: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-460.bz2
Mar 24 11:19:30 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="131" num_updates="15" >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="132" num_updates="1" >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 60: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:19:38 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.132.1): ok (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=60, ref=pe_calc-dc-1269425978-69, seq=840, quorate=0
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:0#011(ha-test1)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:38 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 8: 11 actions in 11 synapses
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1269425978-69) derived from /var/lib/pengine/pe-warn-461.bz2
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 70: notify resDRBD:1_pre_notify_start_0 on ha-test2
Mar 24 11:19:38 ha-test1 cib: [14781]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-70.raw
Mar 24 11:19:38 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 8: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-461.bz2
Mar 24 11:19:38 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:19:38 ha-test1 cib: [14781]: info: write_cib_contents: Wrote version 0.132.0 of the CIB to disk (digest: c9bf29eac5472da64c039e3089cea654)
Mar 24 11:19:38 ha-test1 cib: [14781]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.H3ovPO (digest: /var/lib/heartbeat/crm/cib.BBA0ma)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_start_0 (70) confirmed on ha-test2 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: start resDRBD:0_start_0 on ha-test1 (local)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=9:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_start_0 )
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:28: start
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout)         allow-two-primaries;
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:19:38 ha-test1 crm_attribute: [14834]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_start_0 (call=28, rc=0, cib-update=61, confirmed=true) ok
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_start_0 (9) confirmed on ha-test1 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 69: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=69:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:29: notify
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 71: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=29, rc=0, cib-update=62, confirmed=true) ok
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (69) confirmed on ha-test1 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (71) confirmed on ha-test2 (rc=0)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=10:8:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_30000 )
Mar 24 11:19:38 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:30: monitor
Mar 24 11:19:38 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10)
Mar 24 11:19:38 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 25: master-resDRBD:0=10
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.132.6) : Transient attribute: update
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:19:38 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=30, rc=0, cib-update=63, confirmed=false) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (10) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:39 ha-test1 crmd: [9046]: notice: run_graph: Transition 8 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-461.bz2): Complete
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 8 is now complete
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 64: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=64, ref=pe_calc-dc-1269425979-75, seq=840, quorate=0
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:0#011(Slave -> Master ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:1#011(ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:39 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 9: 25 actions in 25 synapses
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 9 (ref=pe_calc-dc-1269425979-75) derived from /var/lib/pengine/pe-input-334.bz2
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 1: cancel resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: send_direct_ack: ACK'ing resource op resDRBD:0_monitor_30000 from 1:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269425979-77
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269425979-77 from ha-test1
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (1) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=30, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=81:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:31: notify
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 83: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: start resDLM:1_start_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_start_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:32: start
Mar 24 11:19:39 ha-test1 pengine: [9045]: info: process_pe_message: Transition 9: PEngine Input stored in: /var/lib/pengine/pe-input-334.bz2
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (83) confirmed on ha-test2 (rc=0)
Mar 24 11:19:39 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 14931 (0x885f638)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=31, rc=0, cib-update=66, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (81) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: promote resDRBD:0_promote_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_promote_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:33: promote
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout)         allow-two-primaries;
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout) 
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_promote_0 (call=33, rc=0, cib-update=67, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_promote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 82: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=82:9:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:34: notify
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 84: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (84) confirmed on ha-test2 (rc=0)
Mar 24 11:19:39 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:19:39 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 27: master-resDRBD:0=10000
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.132.13) : Transient attribute: update
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:19:39 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=34, rc=0, cib-update=68, confirmed=true) ok
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (82) confirmed on ha-test1 (rc=0)
Mar 24 11:19:39 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_start_0 (call=32, rc=0, cib-update=69, confirmed=true) ok
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_start_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:40 ha-test1 crmd: [9046]: notice: run_graph: Transition 9 (Complete=17, Pending=0, Fired=0, Skipped=6, Incomplete=2, Source=/var/lib/pengine/pe-input-334.bz2): Stopped
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 9 is now complete
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 70: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=70, ref=pe_calc-dc-1269425980-84, seq=840, quorate=0
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:19:40 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:19:40 ha-test1 crmd: [9046]: WARN: destroy_action: Cancelling timer for action 1 (src=129)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 10: 10 actions in 10 synapses
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 10 (ref=pe_calc-dc-1269425980-84) derived from /var/lib/pengine/pe-input-335.bz2
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: monitor resDRBD:0_monitor_20000 on ha-test1 (local)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:10:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_20000 )
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:35: monitor
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 46: monitor resDLM:1_monitor_120000 on ha-test1 (local)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=46:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_monitor_120000 )
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:36: monitor
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 53: start resO2CB:1_start_0 on ha-test1 (local)
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=53:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_start_0 )
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:37: start
Mar 24 11:19:40 ha-test1 pengine: [9045]: info: process_pe_message: Transition 10: PEngine Input stored in: /var/lib/pengine/pe-input-335.bz2
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=36, rc=0, cib-update=71, confirmed=false) ok
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_monitor_120000 (46) confirmed on ha-test1 (rc=0)
Mar 24 11:19:40 ha-test1 o2cb[15017]: INFO: ocfs2 not loaded
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:19:40 INFO: ocfs2 not loaded
Mar 24 11:19:40 ha-test1 o2cb[15017]: INFO: Starting resO2CB:1
Mar 24 11:19:40 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:19:40 INFO: Starting resO2CB:1
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=35, rc=8, cib-update=72, confirmed=false) master
Mar 24 11:19:40 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_20000 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:19:40 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 15089 (0x885a160)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_start_0 (call=37, rc=0, cib-update=73, confirmed=true) ok
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_start_0 (53) confirmed on ha-test1 (rc=0)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 54: monitor resO2CB:1_monitor_120000 on ha-test1 (local)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=54:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_monitor_120000 )
Mar 24 11:19:42 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:38: monitor
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 61: start resFS:1_start_0 on ha-test1 (local)
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=61:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_start_0 )
Mar 24 11:19:42 ha-test1 lrmd: [9043]: info: rsc:resFS:1:39: start
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=38, rc=0, cib-update=74, confirmed=false) ok
Mar 24 11:19:42 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_monitor_120000 (54) confirmed on ha-test1 (rc=0)
Mar 24 11:19:42 ha-test1 Filesystem[15102]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:19:42 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:start:stderr) 2010/03/24_11:19:42 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_start_0 (call=39, rc=0, cib-update=75, confirmed=true) ok
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_start_0 (61) confirmed on ha-test1 (rc=0)
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 62: monitor resFS:1_monitor_120000 on ha-test1 (local)
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=62:10:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_monitor_120000 )
Mar 24 11:19:43 ha-test1 lrmd: [9043]: info: rsc:resFS:1:40: monitor
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=40, rc=0, cib-update=76, confirmed=false) ok
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_monitor_120000 (62) confirmed on ha-test1 (rc=0)
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:19:43 ha-test1 crmd: [9046]: notice: run_graph: Transition 10 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-335.bz2): Complete
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 10 is now complete
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: notify_crmd: Transition 10 status: done - <null>
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:19:43 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:22:42 ha-test1 corosync[9029]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 844: memb=1, new=0, lost=1
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: lost: ha-test2 1550025994
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 844: memb=1, new=0, lost=0
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: ais_mark_unseen_peer_dead: Node ha-test2 was not seen in the previous transition
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: lost
Mar 24 11:22:45 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 844 to 0 children
Mar 24 11:22:45 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:22:45 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:22:45 ha-test1 cib: [9042]: info: ais_dispatch: Membership 844: quorum still lost
Mar 24 11:22:45 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000348(null)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 844: quorum still lost
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now lost (was member)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000348(null)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: erase_node_from_join: Removed node ha-test2 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
Mar 24 11:22:45 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/77, version=0.132.21): ok (rc=0)
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="844" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="840" seen="844" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:22:45 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:22:45 ha-test1 crmd: [9046]: WARN: match_down_event: No match for shutdown action on ha-test2
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: te_update_diff: Stonith/shutdown of ha-test2 not matched
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=ha-test2, magic=NA, cib=0.132.22) : Node failure
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 79: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=79, ref=pe_calc-dc-1269426165-92, seq=844, quorate=0
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:1 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:22:45 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 11: 0 actions in 0 synapses
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 11 (ref=pe_calc-dc-1269426165-92) derived from /var/lib/pengine/pe-warn-462.bz2
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:22:45 ha-test1 crmd: [9046]: notice: run_graph: Transition 11 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-462.bz2): Complete
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 11 is now complete
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: notify_crmd: Transition 11 status: done - <null>
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:22:45 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:22:45 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 11: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-462.bz2
Mar 24 11:22:45 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 848: memb=1, new=0, lost=0
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 848: memb=2, new=1, lost=0
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: member
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: NEW:  ha-test2 1550025994
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 848 to 0 children
Mar 24 11:23:12 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:23:12 ha-test1 cib: [9042]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now member (was lost)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=840 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/80, version=0.132.23): ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=local/crmd/81, version=0.132.24): ok (rc=0)
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: 0x88590d0 Node 1550025994 (ha-test2) born on: 848
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/82, version=0.132.24): ok (rc=0)
Mar 24 11:23:12 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 848 to 0 children
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 cib: [9042]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=resDLM:0_monitor_0, magic=0:7;10:1:7:ddac39ed-6665-4cec-98ce-67e0216b7011, cib=0.132.23) : Resource op removal
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.132.24) : Transient attribute: removal
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 848: quorum still lost
Mar 24 11:23:12 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/84, version=0.132.25): ok (rc=0)
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="848" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="848" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:23:12 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_state_transition: Membership changed: 840 -> 848 - join restart
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 86: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: join_make_offer: Making join offers based on membership 848
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks
Mar 24 11:23:12 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:23:12 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: A new node joined the cluster
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-3: Waiting on 2 outstanding join acks
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_finalize: join-3: Syncing the CIB from ha-test1 to the rest of the cluster
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/89, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/90, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/91, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=ha-test2/crmd/6, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=ha-test2/crmd/7, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-3: Updating node state to member for ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-3: Updating node state to member for ha-test1
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/92, version=0.132.25): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test1']/lrm (origin=local/crmd/94, version=0.132.27): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/96, version=0.132.28): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: crm_update_quorum: Updating quorum status to false (call=98)
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:23:14 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/98, version=0.132.28): ok (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 99: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test1']/lrm": ok (rc=0)
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_update_diff: Detected LRM refresh - 7 resources updated: Skipping all resource events
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:227 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.132.28) : LRM Refresh
Mar 24 11:23:14 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (<null>)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=99, ref=pe_calc-dc-1269426194-104, seq=848, quorate=0
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 100: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=100, ref=pe_calc-dc-1269426194-105, seq=848, quorate=0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:1#011(ha-test2)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: handle_response: pe_calc calculation pe_calc-dc-1269426194-104 is obsolete
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 12: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-463.bz2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:1#011(ha-test2)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:23:14 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 13: 17 actions in 17 synapses
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 13 (ref=pe_calc-dc-1269426194-105) derived from /var/lib/pengine/pe-warn-464.bz2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: monitor resDRBD:1_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDLM:0_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 11: monitor resO2CB:0_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 12: monitor resFS:0_monitor_0 on ha-test2
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 77: notify resDRBD:0_pre_notify_start_0 on ha-test1 (local)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=77:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:14 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:41: notify
Mar 24 11:23:14 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 13: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-464.bz2
Mar 24 11:23:14 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:23:14 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=41, rc=0, cib-update=101, confirmed=true) ok
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_start_0 (77) confirmed on ha-test1 (rc=0)
Mar 24 11:23:14 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_0 (10) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_0 (11) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_0 (12) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_0 (9) confirmed on ha-test2 (rc=0)
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on ha-test2 - no waiting
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 6 fired and confirmed
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:23:15 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: start resDRBD:1_start_0 on ha-test2
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_start_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 78: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=78:13:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:16 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:42: notify
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 79: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:23:16 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=42, rc=0, cib-update=102, confirmed=true) ok
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (78) confirmed on ha-test1 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (79) confirmed on ha-test2 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:23:16 ha-test1 crmd: [9046]: notice: run_graph: Transition 13 (Complete=17, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-464.bz2): Complete
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 13 is now complete
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: notify_crmd: Transition 13 status: done - <null>
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:23:16 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.132.38) : Transient attribute: update
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 103: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.132.39) : Transient attribute: update
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 104: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=104, ref=pe_calc-dc-1269426197-116, seq=848, quorate=0
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test2 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:1#011(Slave -> Master ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:0#011(ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:0#011(ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:0#011(ha-test2)
Mar 24 11:23:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 14: 25 actions in 25 synapses
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 14 (ref=pe_calc-dc-1269426197-116) derived from /var/lib/pengine/pe-input-336.bz2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 5: cancel resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 85: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=85:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:43: notify
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 87: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 43: start resDLM:0_start_0 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269426197-4 from ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (5) confirmed on ha-test2 (rc=0)
Mar 24 11:23:17 ha-test1 pengine: [9045]: info: process_pe_message: Transition 14: PEngine Input stored in: /var/lib/pengine/pe-input-336.bz2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=43, rc=0, cib-update=105, confirmed=true) ok
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (85) confirmed on ha-test1 (rc=0)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (87) confirmed on ha-test2 (rc=0)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: promote resDRBD:1_promote_0 on ha-test2
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_promote_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 86: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=86:14:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:23:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:44: notify
Mar 24 11:23:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 88: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:23:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=44, rc=0, cib-update=106, confirmed=true) ok
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (86) confirmed on ha-test1 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (88) confirmed on ha-test2 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_20000 on ha-test2
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_20000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_start_0 (43) confirmed on ha-test2 (rc=0)
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 44: monitor resDLM:0_monitor_120000 on ha-test2
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:23:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 51: start resO2CB:0_start_0 on ha-test2
Mar 24 11:23:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_120000 (44) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_start_0 (51) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: monitor resO2CB:0_monitor_120000 on ha-test2
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: start resFS:0_start_0 on ha-test2
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_120000 (52) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_start_0 (59) confirmed on ha-test2 (rc=0)
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 60: monitor resFS:0_monitor_120000 on ha-test2
Mar 24 11:23:21 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_120000 (60) confirmed on ha-test2 (rc=0)
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:23:22 ha-test1 crmd: [9046]: notice: run_graph: Transition 14 (Complete=25, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-336.bz2): Complete
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 14 is now complete
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: notify_crmd: Transition 14 status: done - <null>
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:23:22 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:26:12 ha-test1 cib: [9042]: info: cib_stats: Processed 202 operations (1633.00us average, 0% utilization) in the last 10min
Mar 24 11:29:27 ha-test1 logger: Mark:HB_REPORT:1269426564
Mar 24 11:29:30 ha-test1 cibadmin: [16788]: info: Invoked: cibadmin -Ql 
Mar 24 11:29:30 ha-test1 crm_shadow: [16828]: info: Invoked: crm_shadow 
Mar 24 11:29:30 ha-test1 cibadmin: [16829]: info: Invoked: cibadmin -Ql 
Mar 24 11:35:26 ha-test1 logger: Mark:HB_REPORT:1269426925
Mar 24 11:35:29 ha-test1 cibadmin: [18451]: info: Invoked: cibadmin -Ql 
Mar 24 11:35:29 ha-test1 crm_shadow: [18491]: info: Invoked: crm_shadow 
Mar 24 11:35:30 ha-test1 cibadmin: [18492]: info: Invoked: cibadmin -Ql 
Mar 24 11:36:12 ha-test1 cib: [9042]: info: cib_stats: Processed 40 operations (2000.00us average, 0% utilization) in the last 10min
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 107: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=107, ref=pe_calc-dc-1269427102-130, seq=848, quorate=0
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:38:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 15: 0 actions in 0 synapses
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 15 (ref=pe_calc-dc-1269427102-130) derived from /var/lib/pengine/pe-input-337.bz2
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:38:22 ha-test1 crmd: [9046]: notice: run_graph: Transition 15 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-337.bz2): Complete
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 15 is now complete
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: notify_crmd: Transition 15 status: done - <null>
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:38:22 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:38:22 ha-test1 pengine: [9045]: info: process_pe_message: Transition 15: PEngine Input stored in: /var/lib/pengine/pe-input-337.bz2
Mar 24 11:38:42 ha-test1 logger: Mark:HB_REPORT:1269427121
Mar 24 11:38:45 ha-test1 cibadmin: [19775]: info: Invoked: cibadmin -Ql 
Mar 24 11:38:46 ha-test1 crm_shadow: [19815]: info: Invoked: crm_shadow 
Mar 24 11:38:46 ha-test1 cibadmin: [19816]: info: Invoked: cibadmin -Ql 
Mar 24 11:46:12 ha-test1 cib: [9042]: info: cib_stats: Processed 37 operations (1081.00us average, 0% utilization) in the last 10min
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped!
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 108: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=108, ref=pe_calc-dc-1269428002-131, seq=848, quorate=0
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:53:22 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 16: 0 actions in 0 synapses
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 16 (ref=pe_calc-dc-1269428002-131) derived from /var/lib/pengine/pe-input-338.bz2
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:53:22 ha-test1 crmd: [9046]: notice: run_graph: Transition 16 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-338.bz2): Complete
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 16 is now complete
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: notify_crmd: Transition 16 status: done - <null>
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:53:22 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:53:22 ha-test1 pengine: [9045]: info: process_pe_message: Transition 16: PEngine Input stored in: /var/lib/pengine/pe-input-338.bz2
Mar 24 11:56:12 ha-test1 cib: [9042]: info: cib_stats: Processed 33 operations (606.00us average, 0% utilization) in the last 10min
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="132" num_updates="52" >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="133" num_updates="1" >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 109: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:56:17 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.133.1): ok (rc=0)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=109, ref=pe_calc-dc-1269428177-132, seq=848, quorate=0
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:56:17 ha-test1 cib: [22665]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-71.raw
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Demote resDRBD:0#011(Master -> Stopped ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDRBD:0#011(ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resDLM:1#011(ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resO2CB:1#011(ha-test1)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:17 ha-test1 pengine: [9045]: notice: LogActions: Stop resource resFS:1#011(ha-test1)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:17 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 17: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-465.bz2
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 17: 31 actions in 31 synapses
Mar 24 11:56:17 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 17 (ref=pe_calc-dc-1269428177-132) derived from /var/lib/pengine/pe-warn-465.bz2
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 39 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 62 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 84: notify resDRBD:0_pre_notify_demote_0 on ha-test1 (local)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=84:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:45: notify
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 86: notify resDRBD:1_pre_notify_demote_0 on ha-test2
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: stop resFS:1_stop_0 on ha-test1 (local)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=59:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_stop_0 )
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: rsc:resFS:1:46: stop
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=40, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:17 ha-test1 cib: [22665]: info: write_cib_contents: Wrote version 0.133.0 of the CIB to disk (digest: 4ac14c3b34ff566124e9c8cc8bd23cff)
Mar 24 11:56:17 ha-test1 cib: [22665]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.RpSDV6 (digest: /var/lib/heartbeat/crm/cib.4ylhzK)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_demote_0 (86) confirmed on ha-test2 (rc=0)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=45, rc=0, cib-update=110, confirmed=true) ok
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_demote_0 (84) confirmed on ha-test1 (rc=0)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 40 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 37 fired and confirmed
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: demote resDRBD:0_demote_0 on ha-test1 (local)
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_demote_0 )
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:47: demote
Mar 24 11:56:17 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=35, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:17 ha-test1 Filesystem[22667]: INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:56:17 INFO: Running stop for /dev/drbd0 on /nfs
Mar 24 11:56:17 ha-test1 Filesystem[22667]: INFO: Trying to unmount /nfs
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:56:17 INFO: Trying to unmount /nfs
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0: State change failed: (-12) Device is held open by someone
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) Command '
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) drbdsetup
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) secondary
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) ' terminated with exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:17 ERROR: r0: Command output: 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0: State change failed: (-12) Device is held open by someone
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) Command '
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) drbdsetup
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 0
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr)  
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) secondary
Mar 24 11:56:17 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) ' terminated with exit code 11
Mar 24 11:56:17 ha-test1 drbd[22709]: ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Called drbdadm -c /etc/drbd.conf secondary r0
Mar 24 11:56:18 ha-test1 Filesystem[22667]: INFO: unmounted /nfs successfully
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:stop:stderr) 2010/03/24_11:56:18 INFO: unmounted /nfs successfully
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Exit code 11
Mar 24 11:56:18 ha-test1 drbd[22709]: ERROR: r0: Command output: 
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stderr) 2010/03/24_11:56:18 ERROR: r0: Command output: 
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_stop_0 (call=46, rc=0, cib-update=111, confirmed=true) ok
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_stop_0 (59) confirmed on ha-test1 (rc=0)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: stop resO2CB:1_stop_0 on ha-test1 (local)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=52:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_stop_0 )
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:48: stop
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=38, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:56:18 ha-test1 o2cb[22818]: INFO: Stopping resO2CB:1
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:56:18 INFO: Stopping resO2CB:1
Mar 24 11:56:18 ha-test1 o2cb[22818]: INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) 2010/03/24_11:56:18 INFO: Stopping ocfs2_controld.pcmk
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:stop:stderr) /usr/lib/ocf/resource.d//pacemaker/o2cb: line 169: killproc: command not found
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_stop_0 (call=48, rc=0, cib-update=112, confirmed=true) ok
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_stop_0 (52) confirmed on ha-test1 (rc=0)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: stop resDLM:1_stop_0 on ha-test1 (local)
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_stop_0 )
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:49: stop
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:56:18 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=36, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:18 ha-test1 ocfs2_controld[15089]: Error from dlmc_fs_result: Bad file descriptor
Mar 24 11:56:18 ha-test1 ocfs2_controld[15089]: dlmcontrol connection died
Mar 24 11:56:18 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:stop:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:56:19 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:demote:stdout) 
Mar 24 11:56:19 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_stop_0 (call=49, rc=0, cib-update=113, confirmed=true) ok
Mar 24 11:56:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_stop_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:56:19 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 49 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_demote_0 (call=47, rc=0, cib-update=114, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_demote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 38 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 41 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 85: notify resDRBD:0_post_notify_demote_0 on ha-test1 (local)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=85:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:50: notify
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 87: notify resDRBD:1_post_notify_demote_0 on ha-test2
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=50, rc=0, cib-update=115, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_demote_0 (85) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_demote_0 (87) confirmed on ha-test2 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 42 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 79: notify resDRBD:0_pre_notify_stop_0 on ha-test1 (local)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=79:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:51: notify
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 80: notify resDRBD:1_pre_notify_stop_0 on ha-test2
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=51, rc=0, cib-update=116, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_stop_0 (79) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_stop_0 (80) confirmed on ha-test2 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 25 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 14: stop resDRBD:0_stop_0 on ha-test1 (local)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=14:17:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_stop_0 )
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:52: stop
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (-INFINITY)
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 52: master-resDRBD:0=-INFINITY
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.133.12) : Transient attribute: update
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 crm_attribute: [22976]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (<null>)
Mar 24 11:56:20 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent delete 54: node=ha-test1, attr=master-resDRBD:0, id=<n/a>, set=(null), section=status
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.133.13) : Transient attribute: removal
Mar 24 11:56:20 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:stop:stdout) 
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_stop_0 (call=52, rc=0, cib-update=117, confirmed=true) ok
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_stop_0 (14) confirmed on ha-test1 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:1_post_notify_stop_0 on ha-test2
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_stop_0 (81) confirmed on ha-test2 (rc=0)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 30 fired and confirmed
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:20 ha-test1 crmd: [9046]: notice: run_graph: Transition 17 (Complete=30, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-warn-465.bz2): Stopped
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 17 is now complete
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 118: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=118, ref=pe_calc-dc-1269428180-145, seq=848, quorate=0
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: unpack_status: Node ha-test1 is in standby-mode
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is standby
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:0 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Stopped)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:20 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 18: 0 actions in 0 synapses
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 18 (ref=pe_calc-dc-1269428180-145) derived from /var/lib/pengine/pe-warn-466.bz2
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:20 ha-test1 crmd: [9046]: notice: run_graph: Transition 18 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-466.bz2): Complete
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 18 is now complete
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: notify_crmd: Transition 18 status: done - <null>
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:20 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:56:20 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 18: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-466.bz2
Mar 24 11:56:20 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - <cib admin_epoch="0" epoch="133" num_updates="15" >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   <configuration >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     <nodes >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       <node id="ha-test1" >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -           <nvpair value="on" id="nodes-ha-test1-standby" />
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -         </instance_attributes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -       </node>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -     </nodes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: -   </configuration>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: - </cib>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + <cib admin_epoch="0" epoch="134" num_updates="1" >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   <configuration >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: need_abort: Aborting on change to admin_epoch
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     <nodes >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       <node id="ha-test1" >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         <instance_attributes id="nodes-ha-test1" >
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 119: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +           <nvpair value="off" id="nodes-ha-test1-standby" />
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +         </instance_attributes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +       </node>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +     </nodes>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: +   </configuration>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: log_data_element: cib:diff: + </cib>
Mar 24 11:56:25 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=ha-test2/crm_attribute/4, version=0.134.1): ok (rc=0)
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=119, ref=pe_calc-dc-1269428185-146, seq=848, quorate=0
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:0 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:25 ha-test1 cib: [22977]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-72.raw
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:1 cannot run anywhere
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:1 cannot run anywhere
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:1 cannot run anywhere
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:0 on ha-test1
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:0#011(ha-test1)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Stopped)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:25 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Stopped)
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 19: 11 actions in 11 synapses
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 19 (ref=pe_calc-dc-1269428185-146) derived from /var/lib/pengine/pe-warn-467.bz2
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Mar 24 11:56:25 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 70: notify resDRBD:1_pre_notify_start_0 on ha-test2
Mar 24 11:56:25 ha-test1 cib: [22977]: info: write_cib_contents: Wrote version 0.134.0 of the CIB to disk (digest: fe287a4e655a0b728685dfff20428f47)
Mar 24 11:56:25 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 19: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-467.bz2
Mar 24 11:56:25 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:56:25 ha-test1 cib: [22977]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.ad2qpz (digest: /var/lib/heartbeat/crm/cib.lcORwF)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_start_0 (70) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: start resDRBD:0_start_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=9:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_start_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:53: start
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout)         allow-two-primaries;
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:56:26 ha-test1 crm_attribute: [23030]: info: Invoked: crm_attribute -N ha-test1 -n master-resDRBD:0 -l reboot -D 
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:start:stdout) 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_start_0 (call=53, rc=0, cib-update=120, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_start_0 (9) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 69: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=69:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:54: notify
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 71: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=54, rc=0, cib-update=121, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (69) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (71) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=10:19:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_30000 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:55: monitor
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10)
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 58: master-resDRBD:0=10
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.134.6) : Transient attribute: update
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=55, rc=0, cib-update=122, confirmed=false) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (10) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:26 ha-test1 crmd: [9046]: notice: run_graph: Transition 19 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-467.bz2): Complete
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 19 is now complete
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 123: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=123, ref=pe_calc-dc-1269428186-152, seq=848, quorate=0
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Slave ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:0_monitor_30000 (Slave vs. Master)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:0#011(Slave -> Master ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:1#011(ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:26 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 20: 25 actions in 25 synapses
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 20 (ref=pe_calc-dc-1269428186-152) derived from /var/lib/pengine/pe-input-339.bz2
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 1: cancel resDRBD:0_monitor_30000 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: send_direct_ack: ACK'ing resource op resDRBD:0_monitor_30000 from 1:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011: lrm_invoke-lrmd-1269428186-154
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269428186-154 from ha-test1
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_30000 (1) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_30000 (call=55, status=1, cib-update=0, confirmed=true) Cancelled
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 81: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=81:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:56: notify
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 83: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 45: start resDLM:1_start_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=45:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_start_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:57: start
Mar 24 11:56:26 ha-test1 pengine: [9045]: info: process_pe_message: Transition 20: PEngine Input stored in: /var/lib/pengine/pe-input-339.bz2
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDLM:1:start:stderr) dlm_controld.pcmk: no process killed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (83) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 23126 (0x88624c0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=56, rc=0, cib-update=125, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (81) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: promote resDRBD:0_promote_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_promote_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:58: promote
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout)         allow-two-primaries;
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:promote:stdout) 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_promote_0 (call=58, rc=0, cib-update=126, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_promote_0 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 82: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=82:20:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:59: notify
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 84: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (84) confirmed on ha-test2 (rc=0)
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:56:26 ha-test1 attrd: [9044]: info: attrd_perform_update: Sent update 60: master-resDRBD:0=10000
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=ha-test1, magic=NA, cib=0.134.13) : Transient attribute: update
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: update_abort_priority: Abort action done superceeded by restart
Mar 24 11:56:26 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=59, rc=0, cib-update=127, confirmed=true) ok
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (82) confirmed on ha-test1 (rc=0)
Mar 24 11:56:26 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_start_0 (call=57, rc=0, cib-update=128, confirmed=true) ok
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_start_0 (45) confirmed on ha-test1 (rc=0)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:27 ha-test1 crmd: [9046]: notice: run_graph: Transition 20 (Complete=17, Pending=0, Fired=0, Skipped=6, Incomplete=2, Source=/var/lib/pengine/pe-input-339.bz2): Stopped
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 20 is now complete
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 129: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=129, ref=pe_calc-dc-1269428187-161, seq=848, quorate=0
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ha-test2 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ha-test1 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:1 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test2 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:1 ]
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:0: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: native_merge_weights: resDLM:1: Rolling back scores from cloneO2CB
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:0 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:1 with resDLM:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resDRBD:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resO2CB:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resDLM:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resFS:0 on ha-test2
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:0 with resO2CB:0
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Master ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Started ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Started ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:1#011(ha-test1)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Started ha-test2)
Mar 24 11:56:27 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:1#011(ha-test1)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:27 ha-test1 crmd: [9046]: WARN: destroy_action: Cancelling timer for action 1 (src=246)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 21: 10 actions in 10 synapses
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 21 (ref=pe_calc-dc-1269428187-161) derived from /var/lib/pengine/pe-input-340.bz2
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 13: monitor resDRBD:0_monitor_20000 on ha-test1 (local)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=13:21:8:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_monitor_20000 )
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:60: monitor
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 46: monitor resDLM:1_monitor_120000 on ha-test1 (local)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=46:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDLM:1_monitor_120000 )
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: rsc:resDLM:1:61: monitor
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 53: start resO2CB:1_start_0 on ha-test1 (local)
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=53:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_start_0 )
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:62: start
Mar 24 11:56:27 ha-test1 pengine: [9045]: info: process_pe_message: Transition 21: PEngine Input stored in: /var/lib/pengine/pe-input-340.bz2
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDLM:1_monitor_120000 (call=61, rc=0, cib-update=130, confirmed=false) ok
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:1_monitor_120000 (46) confirmed on ha-test1 (rc=0)
Mar 24 11:56:27 ha-test1 o2cb[23213]: INFO: ocfs2 not loaded
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:56:27 INFO: ocfs2 not loaded
Mar 24 11:56:27 ha-test1 o2cb[23213]: INFO: Starting resO2CB:1
Mar 24 11:56:27 ha-test1 lrmd: [9043]: info: RA output: (resO2CB:1:start:stderr) 2010/03/24_11:56:27 INFO: Starting resO2CB:1
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_monitor_20000 (call=60, rc=8, cib-update=131, confirmed=false) master
Mar 24 11:56:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_monitor_20000 (13) confirmed on ha-test1 (rc=0)
Mar 24 11:56:27 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_notify: Enabling node notifications for child 23285 (0x8859930)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_start_0 (call=62, rc=0, cib-update=132, confirmed=true) ok
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_start_0 (53) confirmed on ha-test1 (rc=0)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 54: monitor resO2CB:1_monitor_120000 on ha-test1 (local)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=54:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resO2CB:1_monitor_120000 )
Mar 24 11:56:29 ha-test1 lrmd: [9043]: info: rsc:resO2CB:1:63: monitor
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 61: start resFS:1_start_0 on ha-test1 (local)
Mar 24 11:56:29 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=61:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_start_0 )
Mar 24 11:56:29 ha-test1 lrmd: [9043]: info: rsc:resFS:1:64: start
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resO2CB:1_monitor_120000 (call=63, rc=0, cib-update=133, confirmed=false) ok
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:1_monitor_120000 (54) confirmed on ha-test1 (rc=0)
Mar 24 11:56:30 ha-test1 Filesystem[23298]: INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:56:30 ha-test1 lrmd: [9043]: info: RA output: (resFS:1:start:stderr) 2010/03/24_11:56:30 INFO: Running start for /dev/drbd0 on /nfs
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_start_0 (call=64, rc=0, cib-update=134, confirmed=true) ok
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_start_0 (61) confirmed on ha-test1 (rc=0)
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 62: monitor resFS:1_monitor_120000 on ha-test1 (local)
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=62:21:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resFS:1_monitor_120000 )
Mar 24 11:56:30 ha-test1 lrmd: [9043]: info: rsc:resFS:1:65: monitor
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resFS:1_monitor_120000 (call=65, rc=0, cib-update=135, confirmed=false) ok
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:1_monitor_120000 (62) confirmed on ha-test1 (rc=0)
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:30 ha-test1 crmd: [9046]: notice: run_graph: Transition 21 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-340.bz2): Complete
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 21 is now complete
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: notify_crmd: Transition 21 status: done - <null>
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:30 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:56:52 ha-test1 corosync[9029]:   [TOTEM ] A processor failed, forming new configuration.
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 852: memb=1, new=0, lost=1
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: lost: ha-test2 1550025994
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 852: memb=1, new=0, lost=0
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: ais_mark_unseen_peer_dead: Node ha-test2 was not seen in the previous transition
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: lost
Mar 24 11:56:56 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 852 to 0 children
Mar 24 11:56:56 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:56:56 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:56:56 ha-test1 cib: [9042]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:56:56 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 852: quorum still lost
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now lost (was member)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=lost (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000350(null)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: erase_node_from_join: Removed node ha-test2 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1
Mar 24 11:56:56 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/136, version=0.134.21): ok (rc=0)
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511775 was clipped: 2147483647
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="852" quorate="(null)" expected="3065511775" actual="2" >
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="lost" born="848" seen="852" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:56:56 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:56:56 ha-test1 crmd: [9046]: WARN: match_down_event: No match for shutdown action on ha-test2
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: te_update_diff: Stonith/shutdown of ha-test2 not matched
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=ha-test2, magic=NA, cib=0.134.22) : Node failure
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 138: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=138, ref=pe_calc-dc-1269428216-169, seq=852, quorate=0
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resDRBD:1 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:1#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:56:56 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 22: 0 actions in 0 synapses
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 22 (ref=pe_calc-dc-1269428216-169) derived from /var/lib/pengine/pe-warn-468.bz2
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:56:56 ha-test1 crmd: [9046]: notice: run_graph: Transition 22 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-468.bz2): Complete
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 22 is now complete
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: notify_crmd: Transition 22 status: done - <null>
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:56:56 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:56:56 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 22: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-468.bz2
Mar 24 11:56:56 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 856: memb=1, new=0, lost=0
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: memb: ha-test1 1533248778
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 856: memb=2, new=1, lost=0
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: Node 1550025994/ha-test2 is now: member
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: NEW:  ha-test2 1550025994
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test1 1533248778
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: pcmk_peer_update: MEMB: ha-test2 1550025994
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test1 corosync[9029]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Mar 24 11:57:18 ha-test1 cib: [9042]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 cib: [9042]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 cib: [9042]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: update_member: 0x88590d0 Node 1550025994 (ha-test2) born on: 856
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: ais_status_callback: status: ha-test2 is now member (was lost)
Mar 24 11:57:18 ha-test1 corosync[9029]:   [pcmk  ] info: send_member_notification: Sending membership update 856 to 0 children
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: crm_update_peer: Node ha-test2: id=1550025994 state=member (new) addr=r(0) ip(10.133.99.92)  votes=1 born=848 seen=0 proc=00000000000000000000000000000358(null)
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/139, version=0.134.23): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=local/crmd/140, version=0.134.24): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 corosync[9029]:   [MAIN  ] Completed service synchronization, ready to provide service.
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/141, version=0.134.24): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=ha-test2/ha-test2/(null), version=0.134.25): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="848" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=resDLM:0_monitor_0, magic=0:7;10:13:7:ddac39ed-6665-4cec-98ce-67e0216b7011, cib=0.134.23) : Resource op removal
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.134.24) : Transient attribute: removal
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/transient_attributes": ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: ais_dispatch: Membership 856: quorum still lost
Mar 24 11:57:18 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/143, version=0.134.25): ok (rc=0)
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: crm_int_helper: Conversion of 3065511770 was clipped: 2147483647
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update <nodes id="856" quorate="(null)" expected="3065511770" actual="2" >
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1533248778" uname="ha-test1" state="member" born="840" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.91) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update   <node id="1550025994" uname="ha-test2" state="member" born="856" seen="856" votes="1" processes="78610" addr="r(0) ip(10.133.99.92) " version="1.0.7" />
Mar 24 11:57:18 ha-test1 crmd: [9046]: ERROR: log_data_element: crm_ais_dispatch: Invalid quorum/membership update </nodes>
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_state_transition: Membership changed: 848 -> 856 - join restart
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 145: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: join_make_offer: Making join offers based on membership 856
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks
Mar 24 11:57:18 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: update_dc: Unset DC ha-test1
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: A new node joined the cluster
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: update_dc: Set DC to ha-test1 (3.0.1)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_finalize: join-5: Syncing the CIB from ha-test1 to the rest of the cluster
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/148, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/149, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/150, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/transient_attributes (origin=ha-test2/crmd/7, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=ha-test2/crmd/8, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-5: Updating node state to member for ha-test2
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test2']/lrm (origin=local/crmd/151, version=0.134.25): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test2']/lrm": ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_ack: join-5: Updating node state to member for ha-test1
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ha-test1']/lrm (origin=local/crmd/153, version=0.134.27): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ha-test1']/lrm": ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/155, version=0.134.28): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: crm_update_quorum: Updating quorum status to false (call=157)
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Mar 24 11:57:19 ha-test1 cib: [9042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/157, version=0.134.28): ok (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 158: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=158, ref=pe_calc-dc-1269428239-181, seq=856, quorate=0
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:0 (10000)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:57:19 ha-test1 attrd: [9044]: info: attrd_trigger_update: Sending flush op to all hosts for: master-resDRBD:1 (<null>)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDRBD:1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 1 instances of a possible 2 to master
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: native_color: Resource resDLM:0 cannot run anywhere
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: native_color: Resource resO2CB:0 cannot run anywhere
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: native_color: Resource resFS:0 cannot run anywhere
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (30s) for resDRBD:1 on ha-test2
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Start resDRBD:1#011(ha-test2)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:0#011(Stopped)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:0#011(Stopped)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:0#011(Stopped)
Mar 24 11:57:19 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 23: 17 actions in 17 synapses
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 23 (ref=pe_calc-dc-1269428239-181) derived from /var/lib/pengine/pe-warn-469.bz2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 9: monitor resDRBD:1_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 10: monitor resDLM:0_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 11: monitor resO2CB:0_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 12: monitor resFS:0_monitor_0 on ha-test2
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 77: notify resDRBD:0_pre_notify_start_0 on ha-test1 (local)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=77:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:19 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:66: notify
Mar 24 11:57:19 ha-test1 pengine: [9045]: WARN: process_pe_message: Transition 23: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-469.bz2
Mar 24 11:57:19 ha-test1 pengine: [9045]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Mar 24 11:57:19 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=66, rc=0, cib-update=159, confirmed=true) ok
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_start_0 (77) confirmed on ha-test1 (rc=0)
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
Mar 24 11:57:19 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_0 (11) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_0 (10) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_0 (9) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_0 (12) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 8: probe_complete probe_complete on ha-test2 - no waiting
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 6 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: start resDRBD:1_start_0 on ha-test2
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_start_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 20 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 78: notify resDRBD:0_post_notify_start_0 on ha-test1 (local)
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=78:23:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:20 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:67: notify
Mar 24 11:57:20 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 79: notify resDRBD:1_post_notify_start_0 on ha-test2
Mar 24 11:57:21 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) drbdsetup 0 syncer --set-defaults --create-device --verify-alg=md5 --rate=100M 
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=67, rc=0, cib-update=160, confirmed=true) ok
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_start_0 (78) confirmed on ha-test1 (rc=0)
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_start_0 (79) confirmed on ha-test2 (rc=0)
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:57:21 ha-test1 crmd: [9046]: notice: run_graph: Transition 23 (Complete=17, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-469.bz2): Complete
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 23 is now complete
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: notify_crmd: Transition 23 status: done - <null>
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:57:21 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.134.38) : Transient attribute: update
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 161: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=ha-test2, magic=NA, cib=0.134.39) : Transient attribute: update
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_pe_invoke: Query 162: Requesting the current CIB: S_POLICY_ENGINE
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_pe_invoke_callback: Invoking the PE: query=162, ref=pe_calc-dc-1269428243-192, seq=856, quorate=0
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test1 is online
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resO2CB:0 on ha-test1 to resO2CB:2 (ORPHAN)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_clone: Internally renamed resFS:0 on ha-test1 to resFS:2 (ORPHAN)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: determine_online_status: Node ha-test2 is online
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Master/Slave Set: msDRBD
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Masters: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Slaves: [ ha-test2 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resDLM:0 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resO2CB:0 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_print:  Clone Set: cloneFS
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Started: [ ha-test1 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: short_print:      Stopped: [ resFS:0 ]
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:0: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: resDRBD:1: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneDLM
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: msDRBD: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: native_merge_weights: cloneDLM: Rolling back scores from cloneO2CB
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:0 (Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: Promoting resDRBD:1 (Slave ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: master_color: msDRBD: Promoted 2 instances of a possible 2 to master
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (20s) for resDRBD:1 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: RecurringOp: Cancelling action resDRBD:1_monitor_30000 (Slave vs. Master)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resDLM:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resO2CB:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: RecurringOp:  Start recurring monitor (120s) for resFS:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:0 with resDRBD:1 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:1 with resDLM:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resDRBD:0 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDRBD:0 with resDLM:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDRBD:0 with resDLM:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resDRBD:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:0 with resDLM:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:0 with resO2CB:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resDLM:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resDLM:1 with resO2CB:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resDLM:1 with resO2CB:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resDLM:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:0 with resO2CB:0 on ha-test2
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:0 with resFS:0
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resFS:1 with resO2CB:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resO2CB:1 with resFS:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: find_compatible_child: Colocating resO2CB:1 with resFS:1 on ha-test1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: clone_rsc_order_lh: Interleaving resFS:1 with resO2CB:1
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDRBD:0#011(Master ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Promote resDRBD:1#011(Slave -> Master ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Start resDLM:0#011(ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resDLM:1#011(Started ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Start resO2CB:0#011(ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resO2CB:1#011(Started ha-test1)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Start resFS:0#011(ha-test2)
Mar 24 11:57:23 ha-test1 pengine: [9045]: notice: LogActions: Leave resource resFS:1#011(Started ha-test1)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: unpack_graph: Unpacked transition 24: 25 actions in 25 synapses
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_te_invoke: Processing graph 24 (ref=pe_calc-dc-1269428243-192) derived from /var/lib/pengine/pe-input-341.bz2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 5: cancel resDRBD:1_monitor_30000 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 33 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 47 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 55 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 63 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 85: notify resDRBD:0_pre_notify_promote_0 on ha-test1 (local)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=85:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:23 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:68: notify
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 87: notify resDRBD:1_pre_notify_promote_0 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 43: start resDLM:0_start_0 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1269428243-6 from ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_30000 (5) confirmed on ha-test2 (rc=0)
Mar 24 11:57:23 ha-test1 pengine: [9045]: info: process_pe_message: Transition 24: PEngine Input stored in: /var/lib/pengine/pe-input-341.bz2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=68, rc=0, cib-update=163, confirmed=true) ok
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_pre_notify_promote_0 (85) confirmed on ha-test1 (rc=0)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_pre_notify_promote_0 (87) confirmed on ha-test2 (rc=0)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 34 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 17: promote resDRBD:1_promote_0 on ha-test2
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_promote_0 (17) confirmed on ha-test2 (rc=0)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 35 fired and confirmed
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 86: notify resDRBD:0_post_notify_promote_0 on ha-test1 (local)
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: do_lrm_rsc_op: Performing key=86:24:0:ddac39ed-6665-4cec-98ce-67e0216b7011 op=resDRBD:0_notify_0 )
Mar 24 11:57:23 ha-test1 lrmd: [9043]: info: rsc:resDRBD:0:69: notify
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 88: notify resDRBD:1_post_notify_promote_0 on ha-test2
Mar 24 11:57:23 ha-test1 lrmd: [9043]: info: RA output: (resDRBD:0:notify:stdout) 
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: process_lrm_event: LRM operation resDRBD:0_notify_0 (call=69, rc=0, cib-update=164, confirmed=true) ok
Mar 24 11:57:23 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:0_post_notify_promote_0 (86) confirmed on ha-test1 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_post_notify_promote_0 (88) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 36 fired and confirmed
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 18: monitor resDRBD:1_monitor_20000 on ha-test2
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDRBD:1_monitor_20000 (18) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_start_0 (43) confirmed on ha-test2 (rc=0)
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 44: monitor resDLM:0_monitor_120000 on ha-test2
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 48 fired and confirmed
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 51: start resO2CB:0_start_0 on ha-test2
Mar 24 11:57:24 ha-test1 crmd: [9046]: info: match_graph_event: Action resDLM:0_monitor_120000 (44) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_start_0 (51) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 52: monitor resO2CB:0_monitor_120000 on ha-test2
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 56 fired and confirmed
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 59: start resFS:0_start_0 on ha-test2
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resO2CB:0_monitor_120000 (52) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_start_0 (59) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_rsc_command: Initiating action 60: monitor resFS:0_monitor_120000 on ha-test2
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_pseudo_action: Pseudo action 64 fired and confirmed
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: match_graph_event: Action resFS:0_monitor_120000 (60) confirmed on ha-test2 (rc=0)
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: run_graph: ====================================================
Mar 24 11:57:27 ha-test1 crmd: [9046]: notice: run_graph: Transition 24 (Complete=25, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-341.bz2): Complete
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: te_graph_trigger: Transition 24 is now complete
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: notify_crmd: Transition 24 status: done - <null>
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Mar 24 11:57:27 ha-test1 crmd: [9046]: info: do_state_transition: Starting PEngine Recheck Timer
Mar 24 11:59:03 ha-test1 logger: Mark:HB_REPORT:1269428341
