Feb  4 15:27:18 INACD-DB-SVR2-TEST sshd[9214]: Accepted publickey for root from 172.16.0.20 port 59682 ssh2
Feb  4 15:28:16 INACD-DB-SVR2-TEST mysql[9678]: INFO: MySQL monitor succeded
Feb  4 15:28:16 INACD-DB-SVR2-TEST lrmd: [3652]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:28:16 INFO: MySQL monitor succeded
Feb  4 15:28:43 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Unloading all Corosync service engines.
Feb  4 15:28:43 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: Shuting down Pacemaker
Feb  4 15:28:43 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: stop_child: Sent -15 to crmd: [3655]
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: crm_shutdown: Requesting shutdown
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_shutdown_req: Sending shutdown request to DC: INACD-DB-SVR2-TEST
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: handle_shutdown_request: Creating shutdown request for INACD-DB-SVR2-TEST (state=S_POLICY_ENGINE)
Feb  4 15:28:43 INACD-DB-SVR2-TEST attrd: [3653]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (1265293723)
Feb  4 15:28:43 INACD-DB-SVR2-TEST attrd: [3653]: info: attrd_perform_update: Sent update 86: shutdown=1265293723
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=INACD-DB-SVR2-TEST, magic=NA, cib=0.82.11) : Transient attribute: update
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_pe_invoke: Query 56: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_pe_invoke_callback: Invoking the PE: query=56, ref=pe_calc-dc-1265293723-20, seq=128, quorate=0
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: determine_online_status: Node INACD-DB-SVR2-TEST is shutting down
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: native_print: mysql-master	(ocf::dk:mysql):	Stopped 
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: native_print: mysql-master-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: native_print: mysql-slave	(ocf::dk:mysql):	Started INACD-DB-SVR2-TEST
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: native_print: mysql-slave-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: clone_print:  Clone Set: cl-pingd
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: short_print:      Started: [ INACD-DB-SVR2-TEST ]
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: short_print:      Stopped: [ pingd:0 ]
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: native_print: apcstonith-INACD-DB-SVR2-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: native_print: apcstonith-INACD-DB-SVR1-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: native_merge_weights: mysql-master: Rolling back scores from mysql-master-ip
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: native_merge_weights: mysql-master: Rolling back scores from mysql-slave-ip
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource mysql-master cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: native_merge_weights: mysql-slave: Rolling back scores from mysql-master-ip
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: native_merge_weights: mysql-slave: Rolling back scores from mysql-slave-ip
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource mysql-slave cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource mysql-master-ip cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource mysql-slave-ip cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource pingd:1 cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource pingd:0 cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource apcstonith-INACD-DB-SVR2-TEST cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: native_color: Resource apcstonith-INACD-DB-SVR1-TEST cannot run anywhere
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: stage6: Scheduling Node INACD-DB-SVR2-TEST for shutdown
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Leave resource mysql-master	(Stopped)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Stop resource mysql-master-ip	(INACD-DB-SVR2-TEST)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Stop resource mysql-slave	(INACD-DB-SVR2-TEST)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Stop resource mysql-slave-ip	(INACD-DB-SVR2-TEST)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Leave resource pingd:0	(Stopped)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Stop resource pingd:1	(INACD-DB-SVR2-TEST)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Stop resource apcstonith-INACD-DB-SVR2-TEST	(INACD-DB-SVR2-TEST)
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: notice: LogActions: Stop resource apcstonith-INACD-DB-SVR1-TEST	(INACD-DB-SVR2-TEST)
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: unpack_graph: Unpacked transition 2: 10 actions in 10 synapses
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1265293723-20) derived from /var/lib/pengine/pe-warn-57.bz2
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: te_rsc_command: Initiating action 10: stop mysql-master-ip_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_rsc_op: Performing key=10:2:0:57f3cb55-b788-4142-80d0-b84465332768 op=mysql-master-ip_stop_0 )
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: rsc:mysql-master-ip:21: stop
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: te_rsc_command: Initiating action 11: stop mysql-slave_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_rsc_op: Performing key=11:2:0:57f3cb55-b788-4142-80d0-b84465332768 op=mysql-slave_stop_0 )
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: WARN: process_pe_message: Transition 2: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-57.bz2
Feb  4 15:28:43 INACD-DB-SVR2-TEST pengine: [3654]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: rsc:mysql-slave:22: stop
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: te_rsc_command: Initiating action 12: stop mysql-slave-ip_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_rsc_op: Performing key=12:2:0:57f3cb55-b788-4142-80d0-b84465332768 op=mysql-slave-ip_stop_0 )
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: rsc:mysql-slave-ip:23: stop
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-master-ip_monitor_10000 (call=19, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-slave_monitor_10000 (call=14, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-slave_monitor_180000 (call=15, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-slave-ip_monitor_10000 (call=20, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: te_rsc_command: Initiating action 13: stop pingd:1_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_rsc_op: Performing key=13:2:0:57f3cb55-b788-4142-80d0-b84465332768 op=pingd:1_stop_0 )
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: rsc:pingd:1:24: stop
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation pingd:1_monitor_30000 (call=12, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:28:43 INACD-DB-SVR2-TEST pingd: [3764]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation pingd:1_stop_0 (call=24, rc=0, cib-update=57, confirmed=true) ok
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: match_graph_event: Action pingd:1_stop_0 (13) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:28:43 INACD-DB-SVR2-TEST IPaddr2[9911]: INFO: ip -f inet addr delete 10.2.50.24/24 dev eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST IPaddr2[9915]: INFO: ip -f inet addr delete 10.2.50.240/24 dev eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: RA output: (mysql-master-ip:stop:stderr) 2010/02/04_15:28:43 INFO: ip -f inet addr delete 10.2.50.24/24 dev eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: RA output: (mysql-slave-ip:stop:stderr) 2010/02/04_15:28:43 INFO: ip -f inet addr delete 10.2.50.240/24 dev eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST IPaddr2[9911]: INFO: ip -o -f inet addr show eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST IPaddr2[9915]: INFO: ip -o -f inet addr show eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: RA output: (mysql-master-ip:stop:stderr) 2010/02/04_15:28:43 INFO: ip -o -f inet addr show eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST lrmd: [3652]: info: RA output: (mysql-slave-ip:stop:stderr) 2010/02/04_15:28:43 INFO: ip -o -f inet addr show eth0
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-master-ip_stop_0 (call=21, rc=0, cib-update=58, confirmed=true) ok
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-slave-ip_stop_0 (call=23, rc=0, cib-update=59, confirmed=true) ok
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: match_graph_event: Action mysql-master-ip_stop_0 (10) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:28:43 INACD-DB-SVR2-TEST crmd: [3655]: info: match_graph_event: Action mysql-slave-ip_stop_0 (12) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:28:48 INACD-DB-SVR2-TEST mysql[9914]: INFO: MySQL stopped
Feb  4 15:28:48 INACD-DB-SVR2-TEST lrmd: [3652]: info: RA output: (mysql-slave:stop:stderr) 2010/02/04_15:28:48 INFO: MySQL stopped
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation mysql-slave_stop_0 (call=22, rc=0, cib-update=60, confirmed=true) ok
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: match_graph_event: Action mysql-slave_stop_0 (11) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: te_pseudo_action: Pseudo action 7 fired and confirmed
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: te_rsc_command: Initiating action 18: stop apcstonith-INACD-DB-SVR2-TEST_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_rsc_op: Performing key=18:2:0:57f3cb55-b788-4142-80d0-b84465332768 op=apcstonith-INACD-DB-SVR2-TEST_stop_0 )
Feb  4 15:28:48 INACD-DB-SVR2-TEST lrmd: [3652]: info: rsc:apcstonith-INACD-DB-SVR2-TEST:25: stop
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: te_rsc_command: Initiating action 19: stop apcstonith-INACD-DB-SVR1-TEST_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_rsc_op: Performing key=19:2:0:57f3cb55-b788-4142-80d0-b84465332768 op=apcstonith-INACD-DB-SVR1-TEST_stop_0 )
Feb  4 15:28:48 INACD-DB-SVR2-TEST lrmd: [3652]: info: rsc:apcstonith-INACD-DB-SVR1-TEST:26: stop
Feb  4 15:28:48 INACD-DB-SVR2-TEST lrmd: [10041]: info: Try to stop STONITH resource <rsc_id=apcstonith-INACD-DB-SVR2-TEST> : Device=apcmastersnmp
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 (call=13, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:28:48 INACD-DB-SVR2-TEST lrmd: [10042]: info: Try to stop STONITH resource <rsc_id=apcstonith-INACD-DB-SVR1-TEST> : Device=apcmastersnmp
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_stop_0 (call=25, rc=0, cib-update=61, confirmed=true) ok
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR1-TEST_stop_0 (call=26, rc=0, cib-update=62, confirmed=true) ok
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: match_graph_event: Action apcstonith-INACD-DB-SVR2-TEST_stop_0 (18) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: match_graph_event: Action apcstonith-INACD-DB-SVR1-TEST_stop_0 (19) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: te_crm_command: Executing crm-event (22): do_shutdown on INACD-DB-SVR2-TEST
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: te_crm_command: crm-event (22) is a local shutdown
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: run_graph: ====================================================
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: notice: run_graph: Transition 2 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-57.bz2): Complete
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: te_graph_trigger: Transition 2 is now complete
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_STOPPING [ input=I_STOP cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_dc_release: DC role released
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: pe_connection_destroy: Connection to the Policy Engine released
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_te_control: Transitioner is now inactive
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_te_control: Disconnecting STONITH...
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: tengine_stonith_connection_destroy: Fencing daemon disconnected
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: notice: Not currently connected.
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_lrm_control: Disconnected from the LRM
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_ha_control: Disconnected from OpenAIS
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_cib_control: Disconnecting CIB
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: crmd_cib_connection_destroy: Connection to the CIB terminated...
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: free_mem: Dropping I_RELEASE_SUCCESS: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_dc_release ]
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: cib_process_readwrite: We are now in R/O mode
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: WARN: send_ipc_message: IPC Channel to 3655 is not connected
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: WARN: send_via_callback_channel: Delivery of reply to client 3655/b4d4532f-879e-41f1-8356-b0e82e77ce1e failed
Feb  4 15:28:48 INACD-DB-SVR2-TEST crmd: [3655]: info: do_exit: [crmd] stopped (0)
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: WARN: do_local_notify: A-Sync reply to crmd failed: reply failed
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] info: pcmk_ipc_exit: Client crmd (conn=0x6785a0, async-conn=0x6785a0) left
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: crmd confirmed stopped
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: stop_child: Sent -15 to pengine: [3654]
Feb  4 15:28:48 INACD-DB-SVR2-TEST pengine: [3654]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: pengine confirmed stopped
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: stop_child: Sent -15 to attrd: [3653]
Feb  4 15:28:48 INACD-DB-SVR2-TEST attrd: [3653]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Feb  4 15:28:48 INACD-DB-SVR2-TEST attrd: [3653]: info: attrd_shutdown: Exiting
Feb  4 15:28:48 INACD-DB-SVR2-TEST attrd: [3653]: info: main: Exiting...
Feb  4 15:28:48 INACD-DB-SVR2-TEST attrd: [3653]: info: attrd_cib_connection_destroy: Connection to the CIB terminated...
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] info: pcmk_ipc_exit: Client attrd (conn=0x675010, async-conn=0x675010) left
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: attrd confirmed stopped
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: stop_child: Sent -15 to lrmd: [3652]
Feb  4 15:28:48 INACD-DB-SVR2-TEST lrmd: [3652]: info: lrmd is shutting down
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: lrmd confirmed stopped
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: stop_child: Sent -15 to cib: [3651]
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: WARN: disconnect_cib_client: Disconnecting 4022/cib_callback...
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: WARN: disconnect_cib_client: Disconnecting crm_mon/cib_ro...
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: cib_shutdown: Disconnected 2 clients
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: cib_process_disconnect: All clients disconnected...
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: cib_ha_connection_destroy: Exiting...
Feb  4 15:28:48 INACD-DB-SVR2-TEST cib: [3651]: info: main: Done
Feb  4 15:28:48 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] info: pcmk_ipc_exit: Client cib (conn=0x6775f0, async-conn=0x6775f0) left
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: cib confirmed stopped
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: stop_child: Sent -15 to stonithd: [3650]
Feb  4 15:28:49 INACD-DB-SVR2-TEST stonithd: [3650]: notice: /usr/lib64/heartbeat/stonithd normally quit.
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] info: pcmk_ipc_exit: Client stonithd (conn=0x676c40, async-conn=0x676c40) left
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: stonithd confirmed stopped
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] info: update_member: Node INACD-DB-SVR2-TEST now has process list: 00000000000000000000000000000002 (2)
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [pcmk  ] notice: pcmk_shutdown: Shutdown complete
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: Pacemaker Cluster Manager 1.0.7
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: corosync extended virtual synchrony service
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: corosync configuration service
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: corosync cluster config database access v1.01
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: corosync profile loading service
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Feb  4 15:28:49 INACD-DB-SVR2-TEST corosync[3644]:   [MAIN  ] Corosync Cluster Engine exiting with status -1 at main.c:158.
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Corosync Cluster Engine ('1.2.0'): started and ready to provide service.
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Corosync built-in features: nss rdma
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] Initializing transport (UDP/IP).
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] The network interface [10.250.250.38] is now up.
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: process_ais_conf: Reading configure
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: config_find_init: Local handle: 7739444317642555394 for logging
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: config_find_next: Processing additional logging options...
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: get_config_opt: Found 'off' for option: debug
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: get_config_opt: Defaulting to 'off' for option: to_file
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: config_find_init: Local handle: 5650605097994944515 for service
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: config_find_next: Processing additional service options...
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_startup: CRM: Initialized
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] Logging: Initialized pcmk_startup
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_startup: Service: 9
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_startup: Local hostname: INACD-DB-SVR2-TEST
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_update_nodeid: Local node id: 38
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Creating entry for node 38 born on 0
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: 0x7f11d4001b80 Node 38 now known as INACD-DB-SVR2-TEST (was: (null))
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node INACD-DB-SVR2-TEST now has 1 quorum votes (was 0)
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node 38/INACD-DB-SVR2-TEST is now: member
Feb  4 15:28:56 INACD-DB-SVR2-TEST lrmd: [10068]: info: G_main_add_SignalHandler: Added signal handler for signal 15
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: spawn_child: Forked child 10066 for process stonithd
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: spawn_child: Forked child 10067 for process cib
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: spawn_child: Forked child 10068 for process lrmd
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: spawn_child: Forked child 10069 for process attrd
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: spawn_child: Forked child 10070 for process pengine
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: spawn_child: Forked child 10071 for process crmd
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: Pacemaker Cluster Manager 1.0.7
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: corosync configuration service
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: corosync profile loading service
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 132: memb=0, new=0, lost=0
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 132: memb=1, new=1, lost=0
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: NEW:  INACD-DB-SVR2-TEST 38
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: MEMB: INACD-DB-SVR2-TEST 38
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node INACD-DB-SVR2-TEST now has process list: 00000000000000000000000000013312 (78610)
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb  4 15:28:56 INACD-DB-SVR2-TEST lrmd: [10068]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Feb  4 15:28:56 INACD-DB-SVR2-TEST lrmd: [10068]: info: G_main_add_SignalHandler: Added signal handler for signal 10
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: Invoked: /usr/lib64/heartbeat/attrd 
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: Invoked: /usr/lib64/heartbeat/cib 
Feb  4 15:28:56 INACD-DB-SVR2-TEST lrmd: [10068]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: G_main_add_SignalHandler: Added signal handler for signal 12
Feb  4 15:28:56 INACD-DB-SVR2-TEST lrmd: [10068]: info: Started.
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: main: Starting up
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: crm_cluster_connect: Connecting to OpenAIS
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: init_ais_connection: Creating connection to our AIS plugin
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: G_main_add_TriggerHandler: Added signal manual handler
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: Stack hogger failed 0xffffffff
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: init_ais_connection: AIS connection established
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x672bf0 for attrd/10069
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: get_ais_nodeid: Server details: id=38 uname=INACD-DB-SVR2-TEST cname=pcmk
Feb  4 15:28:56 INACD-DB-SVR2-TEST crmd: [10071]: info: Invoked: /usr/lib64/heartbeat/crmd 
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: crm_new_peer: Node INACD-DB-SVR2-TEST now has id: 38
Feb  4 15:28:56 INACD-DB-SVR2-TEST crmd: [10071]: info: main: CRM Hg Version: d3fa20fc76c7947d6de66db7e52526dc6bd7d782
Feb  4 15:28:56 INACD-DB-SVR2-TEST pengine: [10070]: info: Invoked: /usr/lib64/heartbeat/pengine 
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: crm_new_peer: Node 38 is now known as INACD-DB-SVR2-TEST
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: main: Cluster connection active
Feb  4 15:28:56 INACD-DB-SVR2-TEST crmd: [10071]: info: crmd_init: Starting crmd
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: main: Accepting attribute updates
Feb  4 15:28:56 INACD-DB-SVR2-TEST attrd: [10069]: info: main: Starting mainloop...
Feb  4 15:28:56 INACD-DB-SVR2-TEST crmd: [10071]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: crm_cluster_connect: Connecting to OpenAIS
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: init_ais_connection: Creating connection to our AIS plugin
Feb  4 15:28:56 INACD-DB-SVR2-TEST pengine: [10070]: info: main: Starting pengine
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: init_ais_connection: AIS connection established
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x674860 for stonithd/10066
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: get_ais_nodeid: Server details: id=38 uname=INACD-DB-SVR2-TEST cname=pcmk
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: crm_new_peer: Node INACD-DB-SVR2-TEST now has id: 38
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: crm_new_peer: Node 38 is now known as INACD-DB-SVR2-TEST
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: notice: /usr/lib64/heartbeat/stonithd start up successfully.
Feb  4 15:28:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: startCib: CIB Initialization completed successfully
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: crm_cluster_connect: Connecting to OpenAIS
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: init_ais_connection: Creating connection to our AIS plugin
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: init_ais_connection: AIS connection established
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x6751f0 for cib/10067
Feb  4 15:28:56 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_ipc: Sending membership update 132 to cib
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: get_ais_nodeid: Server details: id=38 uname=INACD-DB-SVR2-TEST cname=pcmk
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: crm_new_peer: Node INACD-DB-SVR2-TEST now has id: 38
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: crm_new_peer: Node 38 is now known as INACD-DB-SVR2-TEST
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: cib_init: Starting cib mainloop
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: ais_dispatch: Membership 132: quorum still lost
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10067]: info: crm_update_peer: Node INACD-DB-SVR2-TEST: id=38 state=member (new) addr=r(0) ip(10.250.250.38)  (new) votes=1 (new) born=0 seen=132 proc=00000000000000000000000000013312 (new)
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10075]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-81.raw
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10075]: info: write_cib_contents: Wrote version 0.82.0 of the CIB to disk (digest: 65df1f2ae3237cb6995a9da0e9b05a51)
Feb  4 15:28:56 INACD-DB-SVR2-TEST cib: [10075]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.kuhFg4 (digest: /var/lib/heartbeat/crm/cib.FdU2b9)
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_cib_control: CIB connection established
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_cluster_connect: Connecting to OpenAIS
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: init_ais_connection: Creating connection to our AIS plugin
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: init_ais_connection: AIS connection established
Feb  4 15:28:57 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_ipc: Recorded connection 0x676180 for crmd/10071
Feb  4 15:28:57 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_ipc: Sending membership update 132 to crmd
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: get_ais_nodeid: Server details: id=38 uname=INACD-DB-SVR2-TEST cname=pcmk
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_new_peer: Node INACD-DB-SVR2-TEST now has id: 38
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_new_peer: Node 38 is now known as INACD-DB-SVR2-TEST
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_ha_control: Connected to the cluster
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_started: Delaying start, CCM (0000000000100000) not connected
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: crmd_init: Starting crmd's mainloop
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_dispatch: Membership 132: quorum still lost
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_peer: Node INACD-DB-SVR2-TEST: id=38 state=member (new) addr=r(0) ip(10.250.250.38)  (new) votes=1 (new) born=0 seen=132 proc=00000000000000000000000000013312 (new)
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_started: Delaying start, Config not read (0000000000000040)
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_started: Delaying start, Config not read (0000000000000040)
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: config_query_callback: Checking for expired actions every 900000ms
Feb  4 15:28:57 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_expected_votes: Expected quorum votes 1024 -> 2
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_started: The local CRM is operational
Feb  4 15:28:57 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Feb  4 15:28:58 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_dispatch: Membership 132: quorum still lost
Feb  4 15:29:01 INACD-DB-SVR2-TEST attrd: [10069]: info: cib_connect: Connected to the CIB after 1 signon attempts
Feb  4 15:29:01 INACD-DB-SVR2-TEST attrd: [10069]: info: cib_connect: Sending full refresh
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 136: memb=1, new=0, lost=0
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: memb: INACD-DB-SVR2-TEST 38
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 136: memb=2, new=1, lost=0
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Creating entry for node 36 born on 136
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node 36/unknown is now: member
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: NEW:  .pending. 36
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: MEMB: .pending. 36
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: MEMB: INACD-DB-SVR2-TEST 38
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: send_member_notification: Sending membership update 136 to 2 children
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: 0x7f11d4001b80 Node 38 ((null)) born on: 136
Feb  4 15:29:40 INACD-DB-SVR2-TEST cib: [10067]: notice: ais_dispatch: Membership 136: quorum acquired
Feb  4 15:29:40 INACD-DB-SVR2-TEST cib: [10067]: info: crm_new_peer: Node <null> now has id: 36
Feb  4 15:29:40 INACD-DB-SVR2-TEST cib: [10067]: info: crm_update_peer: Node (null): id=36 state=member (new) addr=r(0) ip(10.250.250.36)  votes=0 born=0 seen=136 proc=00000000000000000000000000000000
Feb  4 15:29:40 INACD-DB-SVR2-TEST crmd: [10071]: notice: ais_dispatch: Membership 136: quorum acquired
Feb  4 15:29:40 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_new_peer: Node <null> now has id: 36
Feb  4 15:29:40 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_peer: Node (null): id=36 state=member (new) addr=r(0) ip(10.250.250.36)  votes=0 born=0 seen=136 proc=00000000000000000000000000000000
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb  4 15:29:40 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_get_peer: Node 36 is now known as INACD-DB-SVR1-TEST
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: 0x676c00 Node 36 (INACD-DB-SVR1-TEST) born on: 136
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: 0x676c00 Node 36 now known as INACD-DB-SVR1-TEST (was: (null))
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node INACD-DB-SVR1-TEST now has process list: 00000000000000000000000000013312 (78610)
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node INACD-DB-SVR1-TEST now has 1 quorum votes (was 0)
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: send_member_notification: Sending membership update 136 to 2 children
Feb  4 15:29:40 INACD-DB-SVR2-TEST cib: [10067]: info: ais_dispatch: Membership 136: quorum retained
Feb  4 15:29:40 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_dispatch: Membership 136: quorum retained
Feb  4 15:29:40 INACD-DB-SVR2-TEST cib: [10067]: info: crm_get_peer: Node 36 is now known as INACD-DB-SVR1-TEST
Feb  4 15:29:40 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_peer: Node INACD-DB-SVR1-TEST: id=36 state=member addr=r(0) ip(10.250.250.36)  votes=1 (new) born=136 seen=136 proc=00000000000000000000000000013312 (new)
Feb  4 15:29:40 INACD-DB-SVR2-TEST cib: [10067]: info: crm_update_peer: Node INACD-DB-SVR1-TEST: id=36 state=member addr=r(0) ip(10.250.250.36)  votes=1 (new) born=136 seen=136 proc=00000000000000000000000000013312 (new)
Feb  4 15:29:40 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped!
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_release: DC role released
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_control: Transitioner is now inactive
Feb  4 15:29:58 INACD-DB-SVR2-TEST crmd: [10071]: info: update_dc: Set DC to INACD-DB-SVR1-TEST (3.0.1)
Feb  4 15:29:59 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=INACD-DB-SVR1-TEST/crmd/18, version=0.82.0): ok (rc=0)
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: update_attrd: Connecting to attrd...
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for terminate
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for shutdown
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR2-TEST']/transient_attributes": ok (rc=0)
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR2-TEST']/lrm": ok (rc=0)
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: crm_new_peer: Node INACD-DB-SVR1-TEST now has id: 36
Feb  4 15:29:59 INACD-DB-SVR2-TEST attrd: [10069]: info: crm_new_peer: Node 36 is now known as INACD-DB-SVR1-TEST
Feb  4 15:29:59 INACD-DB-SVR2-TEST cib: [10082]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-82.raw
Feb  4 15:29:59 INACD-DB-SVR2-TEST cib: [10082]: info: write_cib_contents: Wrote version 0.83.0 of the CIB to disk (digest: f284fd626f0191d139243c0824cfd5c3)
Feb  4 15:29:59 INACD-DB-SVR2-TEST cib: [10082]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.Gttzto (digest: /var/lib/heartbeat/crm/cib.DReRBN)
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=12:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-master_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-master:2: monitor
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=13:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-master-ip_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-master-ip:3: monitor
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=14:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-slave:4: monitor
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=15:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave-ip_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-slave-ip:5: monitor
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=16:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=pingd:1_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:pingd:1:6: monitor
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-master_monitor_0 (call=2, rc=7, cib-update=10, confirmed=true) not running
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: notice: lrmd_rsc_new(): No lrm_rprovider field in message
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=17:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=apcstonith-INACD-DB-SVR2-TEST_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST lrmd: [10068]: notice: lrmd_rsc_new(): No lrm_rprovider field in message
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=18:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=apcstonith-INACD-DB-SVR1-TEST_monitor_0 )
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation pingd:1_monitor_0 (call=6, rc=7, cib-update=11, confirmed=true) not running
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-master-ip_monitor_0 (call=3, rc=7, cib-update=12, confirmed=true) not running
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave_monitor_0 (call=4, rc=7, cib-update=13, confirmed=true) not running
Feb  4 15:29:59 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave-ip_monitor_0 (call=5, rc=7, cib-update=14, confirmed=true) not running
Feb  4 15:30:00 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:apcstonith-INACD-DB-SVR2-TEST:7: monitor
Feb  4 15:30:00 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:apcstonith-INACD-DB-SVR1-TEST:8: monitor
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_monitor_0 (call=7, rc=7, cib-update=15, confirmed=true) not running
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR1-TEST_monitor_0 (call=8, rc=7, cib-update=16, confirmed=true) not running
Feb  4 15:30:00 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for probe_complete
Feb  4 15:30:00 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb  4 15:30:00 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_perform_update: Sent update 11: probe_complete=true
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=8:2:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave_start_0 )
Feb  4 15:30:00 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-slave:9: start
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=21:2:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=apcstonith-INACD-DB-SVR1-TEST_start_0 )
Feb  4 15:30:00 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:apcstonith-INACD-DB-SVR1-TEST:10: start
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=13:2:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=pingd:1_start_0 )
Feb  4 15:30:00 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:pingd:1:11: start
Feb  4 15:30:00 INACD-DB-SVR2-TEST lrmd: [10165]: info: Try to start STONITH resource <rsc_id=apcstonith-INACD-DB-SVR1-TEST> : Device=apcmastersnmp
Feb  4 15:30:00 INACD-DB-SVR2-TEST pingd: [10173]: info: Invoked: /usr/lib64/heartbeat/pingd -D -p /var/run//pingd-pingd -a pingd -d 10 -i 5 -h 10.2.50.11 -h 10.2.50.40 -h 10.2.50.8 
Feb  4 15:30:00 INACD-DB-SVR2-TEST pingd: [10178]: info: main: Starting pingd
Feb  4 15:30:00 INACD-DB-SVR2-TEST su: (to mysql) root on none
Feb  4 15:30:00 INACD-DB-SVR2-TEST pingd: [10178]: info: attrd_lazy_update: Connecting to cluster... 5 retries remaining
Feb  4 15:30:00 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for pingd
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation pingd:1_start_0 (call=11, rc=0, cib-update=17, confirmed=true) ok
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=14:2:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=pingd:1_monitor_30000 )
Feb  4 15:30:00 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation pingd:1_monitor_30000 (call=12, rc=0, cib-update=18, confirmed=false) ok
Feb  4 15:30:01 INACD-DB-SVR2-TEST stonithd: [10066]: info: apcstonith-INACD-DB-SVR1-TEST stonith resource started
Feb  4 15:30:01 INACD-DB-SVR2-TEST lrmd: [10068]: debug: stonithRA plugin: provider attribute is not needed and will be ignored.
Feb  4 15:30:01 INACD-DB-SVR2-TEST lrmd: [10068]: WARN: stonithRA plugin: no short description in apcmastersnmp's metadata.
Feb  4 15:30:01 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR1-TEST_start_0 (call=10, rc=0, cib-update=19, confirmed=true) ok
Feb  4 15:30:06 INACD-DB-SVR2-TEST mysql[10164]: INFO: MySQL started
Feb  4 15:30:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:start:stderr) 2010/02/04_15:30:06 INFO: MySQL started
Feb  4 15:30:06 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave_start_0 (call=9, rc=0, cib-update=20, confirmed=true) ok
Feb  4 15:30:06 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=9:2:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave_monitor_10000 )
Feb  4 15:30:06 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=10:2:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave_monitor_180000 )
Feb  4 15:30:06 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave_monitor_10000 (call=13, rc=0, cib-update=21, confirmed=false) ok
Feb  4 15:30:06 INACD-DB-SVR2-TEST mysql[10276]: INFO: MySQL monitor succeded
Feb  4 15:30:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:30:06 INFO: MySQL monitor succeded
Feb  4 15:30:06 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave_monitor_180000 (call=14, rc=0, cib-update=22, confirmed=false) ok
Feb  4 15:30:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd (3)
Feb  4 15:30:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_perform_update: Sent update 14: pingd=3
Feb  4 15:30:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=18:3:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave-ip_start_0 )
Feb  4 15:30:10 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-slave-ip:15: start
Feb  4 15:30:10 INACD-DB-SVR2-TEST IPaddr2[10293]: INFO: ip -f inet addr add 10.2.50.240/24 brd 10.2.50.255 dev eth0
Feb  4 15:30:10 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave-ip:start:stderr) 2010/02/04_15:30:10 INFO: ip -f inet addr add 10.2.50.240/24 brd 10.2.50.255 dev eth0
Feb  4 15:30:10 INACD-DB-SVR2-TEST IPaddr2[10293]: INFO: ip link set eth0 up
Feb  4 15:30:10 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave-ip:start:stderr) 2010/02/04_15:30:10 INFO: ip link set eth0 up
Feb  4 15:30:10 INACD-DB-SVR2-TEST IPaddr2[10293]: INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-10.2.50.240 eth0 10.2.50.240 auto not_used not_used
Feb  4 15:30:10 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave-ip:start:stderr) 2010/02/04_15:30:10 INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-10.2.50.240 eth0 10.2.50.240 auto not_used not_used
Feb  4 15:30:10 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave-ip_start_0 (call=15, rc=0, cib-update=23, confirmed=true) ok
Feb  4 15:30:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=19:3:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=mysql-slave-ip_monitor_10000 )
Feb  4 15:30:10 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-slave-ip_monitor_10000 (call=16, rc=0, cib-update=24, confirmed=false) ok
Feb  4 15:30:16 INACD-DB-SVR2-TEST crm_shadow: [10406]: info: Invoked: crm_shadow 
Feb  4 15:30:16 INACD-DB-SVR2-TEST cibadmin: [10407]: info: Invoked: cibadmin -Ql 
Feb  4 15:30:33 INACD-DB-SVR2-TEST ntpd[3232]: Deleting interface #6 eth0, 10.2.50.24#123, interface stats: received=0, sent=0, dropped=0, active_time=300 secs
Feb  4 15:31:13 INACD-DB-SVR2-TEST sshd[10651]: Accepted publickey for root from 10.2.20.85 port 49890 ssh2
Feb  4 15:31:26 INACD-DB-SVR2-TEST sshd[10724]: Accepted publickey for root from 10.2.20.85 port 49894 ssh2
Feb  4 15:33:06 INACD-DB-SVR2-TEST mysql[11154]: INFO: MySQL monitor succeded
Feb  4 15:33:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:33:06 INFO: MySQL monitor succeded
Feb  4 15:34:54 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for fail-count-mysql-master-ip
Feb  4 15:35:10 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for last-failure-mysql-master-ip
Feb  4 15:35:28 INACD-DB-SVR2-TEST syslog-ng[2014]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=1154', processed='center(received)=1073', processed='destination(newsnotice)=0', processed='destination(acpid)=1', processed='destination(firewall)=0', processed='destination(null)=3', processed='destination(mail)=2', processed='destination(mailinfo)=2', processed='destination(console)=11', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=1067', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=0', processed='destination(mailerr)=0', processed='destination(xconsole)=11', processed='destination(warn)=57', processed='source(src)=1073'
Feb  4 15:36:06 INACD-DB-SVR2-TEST mysql[11882]: INFO: MySQL monitor succeded
Feb  4 15:36:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:36:06 INFO: MySQL monitor succeded
Feb  4 15:37:20 INACD-DB-SVR2-TEST sshd[12181]: Accepted publickey for root from 172.16.0.20 port 50637 ssh2
Feb  4 15:38:56 INACD-DB-SVR2-TEST cib: [10067]: info: cib_stats: Processed 92 operations (108.00us average, 0% utilization) in the last 10min
Feb  4 15:39:06 INACD-DB-SVR2-TEST mysql[12655]: INFO: MySQL monitor succeded
Feb  4 15:39:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:39:06 INFO: MySQL monitor succeded
Feb  4 15:39:15 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for fail-count-mysql-master
Feb  4 15:39:35 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for last-failure-mysql-master
Feb  4 15:40:56 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for fail-count-pingd:0
Feb  4 15:40:56 INACD-DB-SVR2-TEST attrd: [10069]: info: find_hash_entry: Creating hash entry for last-failure-pingd:0
Feb  4 15:42:06 INACD-DB-SVR2-TEST mysql[13383]: INFO: MySQL monitor succeded
Feb  4 15:42:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:42:06 INFO: MySQL monitor succeded
Feb  4 15:43:56 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=28:12:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=apcstonith-INACD-DB-SVR2-TEST_start_0 )
Feb  4 15:43:56 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:apcstonith-INACD-DB-SVR2-TEST:17: start
Feb  4 15:43:56 INACD-DB-SVR2-TEST lrmd: [13830]: info: Try to start STONITH resource <rsc_id=apcstonith-INACD-DB-SVR2-TEST> : Device=apcmastersnmp
Feb  4 15:43:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: apcstonith-INACD-DB-SVR2-TEST stonith resource started
Feb  4 15:43:56 INACD-DB-SVR2-TEST lrmd: [10068]: debug: stonithRA plugin: provider attribute is not needed and will be ignored.
Feb  4 15:43:56 INACD-DB-SVR2-TEST lrmd: [10068]: WARN: stonithRA plugin: no short description in apcmastersnmp's metadata.
Feb  4 15:43:56 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_start_0 (call=17, rc=0, cib-update=25, confirmed=true) ok
Feb  4 15:43:56 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=29:12:0:a0caa6eb-4af2-4de0-8286-92abff7d2522 op=apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 )
Feb  4 15:43:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: crm_new_peer: Node INACD-DB-SVR1-TEST now has id: 36
Feb  4 15:43:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: crm_new_peer: Node 36 is now known as INACD-DB-SVR1-TEST
Feb  4 15:43:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: stonith_operate_locally::2713: sending fencing op RESET for INACD-DB-SVR1-TEST to apcstonith-INACD-DB-SVR2-TEST (apcmastersnmp) (pid=13833)
Feb  4 15:43:56 INACD-DB-SVR2-TEST stonithd: [10066]: info: Node INACD-DB-SVR2-TEST try to help node INACD-DB-SVR1-TEST to fence node INACD-DB-SVR1-TEST.
Feb  4 15:43:57 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Down
Feb  4 15:43:57 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 (call=18, rc=0, cib-update=26, confirmed=false) ok
Feb  4 15:44:02 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] A processor failed, forming new configuration.
Feb  4 15:44:03 INACD-DB-SVR2-TEST stonithd: [10066]: info: fenced node INACD-DB-SVR1-TEST (the requester): result=SUCCEEDED.
Feb  4 15:44:04 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Up, 100 Mbps full duplex, receive & transmit flow control ON
Feb  4 15:44:06 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Down
Feb  4 15:44:09 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 140: memb=1, new=0, lost=1
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: memb: INACD-DB-SVR2-TEST 38
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: lost: INACD-DB-SVR1-TEST 36
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 140: memb=1, new=0, lost=0
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: MEMB: INACD-DB-SVR2-TEST 38
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: ais_mark_unseen_peer_dead: Node INACD-DB-SVR1-TEST was not seen in the previous transition
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node 36/INACD-DB-SVR1-TEST is now: lost
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: send_member_notification: Sending membership update 140 to 2 children
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: notice: ais_dispatch: Membership 140: quorum lost
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_peer: Node INACD-DB-SVR1-TEST: id=36 state=lost (new) addr=r(0) ip(10.250.250.36)  votes=1 born=136 seen=136 proc=00000000000000000000000000013312
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: notice: ais_dispatch: Membership 140: quorum lost
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: WARN: check_dead_member: Our DC node (INACD-DB-SVR1-TEST) left the cluster
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: crm_update_peer: Node INACD-DB-SVR1-TEST: id=36 state=lost (new) addr=r(0) ip(10.250.250.36)  votes=1 born=136 seen=136 proc=00000000000000000000000000013312
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: update_dc: Unset DC INACD-DB-SVR1-TEST
Feb  4 15:44:09 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_control: Registering TE UUID: c61f0d23-047e-464f-8fe6-cb76fa9a9b5a
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: WARN: cib_client_add_notify_callback: Callback already present
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: set_graph_functions: Setting custom graph functions
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_takeover: Taking over DC status for this partition
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_readwrite: We are now in R/W mode
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/28, version=0.83.57): ok (rc=0)
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/29, version=0.83.57): ok (rc=0)
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/31, version=0.83.57): ok (rc=0)
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: join_make_offer: Making join offers based on membership 140
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_dispatch: Membership 140: quorum still lost
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/33, version=0.83.57): ok (rc=0)
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: config_query_callback: Checking for expired actions every 900000ms
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: update_dc: Set DC to INACD-DB-SVR2-TEST (3.0.1)
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_dispatch: Membership 140: quorum still lost
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/36, version=0.83.57): ok (rc=0)
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_finalize: join-1: Syncing the CIB from INACD-DB-SVR2-TEST to the rest of the cluster
Feb  4 15:44:09 INACD-DB-SVR2-TEST crmd: [10071]: info: te_connect_stonith: Attempting connection to fencing daemon...
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/39, version=0.83.57): ok (rc=0)
Feb  4 15:44:09 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/40, version=0.83.57): ok (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: te_connect_stonith: Connected
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/41, version=0.83.57): ok (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST lrmd: [10068]: debug: stonithRA plugin: provider attribute is not needed and will be ignored.
Feb  4 15:44:10 INACD-DB-SVR2-TEST lrmd: [10068]: WARN: stonithRA plugin: no short description in apcmastersnmp's metadata.
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_ack: join-1: Updating node state to member for INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR2-TEST']/lrm (origin=local/crmd/42, version=0.83.58): ok (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR2-TEST']/lrm": ok (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_quorum: Updating quorum status to false (call=46)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 47: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-mysql-master (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/44, version=0.83.59): ok (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: WARN: match_down_event: No match for shutdown action on INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: te_update_diff: Stonith/shutdown of INACD-DB-SVR1-TEST not matched
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:191 - Triggered transition abort (complete=1, tag=node_state, id=INACD-DB-SVR1-TEST, magic=NA, cib=0.83.60) : Node failure
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 48: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [10067]: info: log_data_element: cib:diff: - <cib have-quorum="1" dc-uuid="INACD-DB-SVR1-TEST" admin_epoch="0" epoch="83" num_updates="60" />
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [10067]: info: log_data_element: cib:diff: + <cib have-quorum="0" dc-uuid="INACD-DB-SVR2-TEST" admin_epoch="0" epoch="84" num_updates="1" />
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/46, version=0.84.1): ok (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: need_abort: Aborting on change to have-quorum
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 49: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-mysql-master (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke_callback: Invoking the PE: query=49, ref=pe_calc-dc-1265294650-14, seq=140, quorate=0
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-mysql-master-ip (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-mysql-master-ip (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd (3)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-pingd:0 (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-pingd:0 (<null>)
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [13899]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-83.raw
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: determine_online_status_fencing: Node INACD-DB-SVR1-TEST (INACD-DB-SVR1-TEST) is un-expectedly down
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status_fencing: 	ha_state=active, ccm_state=false, crm_state=online, join_state=pending, expected=member
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: determine_online_status: Node INACD-DB-SVR1-TEST is unclean
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: unpack_rsc_op: Processing failed op mysql-master_monitor_10000 on INACD-DB-SVR1-TEST: unknown error (1)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: unpack_rsc_op: Processing failed op mysql-master_monitor_180000 on INACD-DB-SVR1-TEST: unknown error (1)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: unpack_rsc_op: Processing failed op mysql-master-ip_monitor_10000 on INACD-DB-SVR1-TEST: unknown error (1)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: unpack_rsc_op: Processing failed op mysql-master-ip_stop_0 on INACD-DB-SVR1-TEST: unknown error (1)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: unpack_rsc_op: Processing failed op pingd:0_monitor_30000 on INACD-DB-SVR1-TEST: unknown error (1)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: unpack_rsc_op: Remapping pingd:0_stop_0 (rc=-2) on INACD-DB-SVR1-TEST to an ERROR
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: unpack_rsc_op: Processing failed op pingd:0_stop_0 on INACD-DB-SVR1-TEST: unknown exec error (-2)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR2-TEST is online
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: native_add_running: Resource stonith::apcmastersnmp:apcstonith-INACD-DB-SVR2-TEST appears to be active on 2 nodes.
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information.
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master	(ocf::dk:mysql):	Started INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave	(ocf::dk:mysql):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: clone_print:  Clone Set: cl-pingd
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Started: [ INACD-DB-SVR2-TEST ]
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [13899]: info: write_cib_contents: Wrote version 0.84.0 of the CIB to disk (digest: bcf823deac8ec334dd49a2d87cd4e8c7)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Stopped: [ pingd:0 ]
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR2-TEST	(stonith:apcmastersnmp) Started 
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: 	0 : INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: 	1 : INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR1-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: get_failcount: mysql-master has failed 2 times on INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: common_apply_stickiness: Forcing mysql-master away from INACD-DB-SVR1-TEST after 2 failures (max=2)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: get_failcount: mysql-master-ip has failed 1000000 times on INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: get_failcount: cl-pingd has failed 1000000 times on INACD-DB-SVR1-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST cib: [13899]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.hDIeGe (digest: /var/lib/heartbeat/crm/cib.sAOb1t)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_merge_weights: mysql-master: Rolling back scores from mysql-master-ip
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_merge_weights: mysql-master: Rolling back scores from mysql-slave-ip
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: native_color: Resource mysql-master cannot run anywhere
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: native_color: Resource pingd:0 cannot run anywhere
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Action mysql-master_stop_0 on INACD-DB-SVR1-TEST is unrunnable (offline)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Marking node INACD-DB-SVR1-TEST unclean
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Action mysql-master-ip_stop_0 on INACD-DB-SVR1-TEST is unrunnable (offline)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Marking node INACD-DB-SVR1-TEST unclean
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (10s) for mysql-master-ip on INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Action pingd:0_stop_0 on INACD-DB-SVR1-TEST is unrunnable (offline)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Marking node INACD-DB-SVR1-TEST unclean
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: native_create_actions: Attempting recovery of resource apcstonith-INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Action apcstonith-INACD-DB-SVR2-TEST_stop_0 on INACD-DB-SVR1-TEST is unrunnable (offline)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: custom_action: Marking node INACD-DB-SVR1-TEST unclean
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (3600s) for apcstonith-INACD-DB-SVR2-TEST on INACD-DB-SVR2-TEST
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: WARN: stage6: Scheduling Node INACD-DB-SVR1-TEST for STONITH
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_stop_constraints: mysql-master_stop_0 is implicit after INACD-DB-SVR1-TEST is fenced
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_stop_constraints: mysql-master-ip_stop_0 is implicit after INACD-DB-SVR1-TEST is fenced
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_stop_constraints: pingd:0_stop_0 is implicit after INACD-DB-SVR1-TEST is fenced
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_start_constraints: Ordering pingd:1_start_0 after INACD-DB-SVR1-TEST recovery
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: native_stop_constraints: apcstonith-INACD-DB-SVR2-TEST_stop_0 is implicit after INACD-DB-SVR1-TEST is fenced
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Stop resource mysql-master	(INACD-DB-SVR1-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Move resource mysql-master-ip	(Started INACD-DB-SVR1-TEST -> INACD-DB-SVR2-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Stop resource pingd:0	(INACD-DB-SVR1-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:1	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Move resource apcstonith-INACD-DB-SVR2-TEST	(Started INACD-DB-SVR1-TEST -> INACD-DB-SVR2-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR1-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: unpack_graph: Unpacked transition 0: 15 actions in 15 synapses
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1265294650-14) derived from /var/lib/pengine/pe-error-2.bz2
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 24 fired and confirmed
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 25: stop apcstonith-INACD-DB-SVR2-TEST_stop_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=25:0:0:c61f0d23-047e-464f-8fe6-cb76fa9a9b5a op=apcstonith-INACD-DB-SVR2-TEST_stop_0 )
Feb  4 15:44:10 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:apcstonith-INACD-DB-SVR2-TEST:19: stop
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 (call=18, status=1, cib-update=0, confirmed=true) Cancelled
Feb  4 15:44:10 INACD-DB-SVR2-TEST lrmd: [13900]: info: Try to stop STONITH resource <rsc_id=apcstonith-INACD-DB-SVR2-TEST> : Device=apcmastersnmp
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_stop_0 (call=19, rc=0, cib-update=50, confirmed=true) ok
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action apcstonith-INACD-DB-SVR2-TEST_stop_0 (25) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 26: start apcstonith-INACD-DB-SVR2-TEST_start_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:44:10 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=26:0:0:c61f0d23-047e-464f-8fe6-cb76fa9a9b5a op=apcstonith-INACD-DB-SVR2-TEST_start_0 )
Feb  4 15:44:10 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:apcstonith-INACD-DB-SVR2-TEST:20: start
Feb  4 15:44:10 INACD-DB-SVR2-TEST lrmd: [13902]: info: Try to start STONITH resource <rsc_id=apcstonith-INACD-DB-SVR2-TEST> : Device=apcmastersnmp
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: ERROR: process_pe_message: Transition 0: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-2.bz2
Feb  4 15:44:10 INACD-DB-SVR2-TEST pengine: [10070]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Feb  4 15:44:11 INACD-DB-SVR2-TEST stonithd: [10066]: info: apcstonith-INACD-DB-SVR2-TEST stonith resource started
Feb  4 15:44:11 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_start_0 (call=20, rc=0, cib-update=51, confirmed=true) ok
Feb  4 15:44:11 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action apcstonith-INACD-DB-SVR2-TEST_start_0 (26) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:44:11 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 2: monitor apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 on INACD-DB-SVR2-TEST (local)
Feb  4 15:44:11 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=2:0:0:c61f0d23-047e-464f-8fe6-cb76fa9a9b5a op=apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 )
Feb  4 15:44:11 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
Feb  4 15:44:11 INACD-DB-SVR2-TEST crmd: [10071]: info: te_fence_node: Executing reboot fencing operation (31) on INACD-DB-SVR1-TEST (timeout=60000)
Feb  4 15:44:11 INACD-DB-SVR2-TEST stonithd: [10066]: info: client tengine [pid: 10071] requests a STONITH operation RESET on node INACD-DB-SVR1-TEST
Feb  4 15:44:11 INACD-DB-SVR2-TEST stonithd: [10066]: info: stonith_operate_locally::2713: sending fencing op RESET for INACD-DB-SVR1-TEST to apcstonith-INACD-DB-SVR2-TEST (apcmastersnmp) (pid=13905)
Feb  4 15:44:11 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Down
Feb  4 15:44:12 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 (call=21, rc=0, cib-update=52, confirmed=false) ok
Feb  4 15:44:12 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action apcstonith-INACD-DB-SVR2-TEST_monitor_3600000 (2) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST stonithd: [10066]: info: Succeeded to STONITH the node INACD-DB-SVR1-TEST: optype=RESET. whodoit: INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: tengine_stonith_callback: call=13905, optype=1, node_name=INACD-DB-SVR1-TEST, result=0, node_list=INACD-DB-SVR2-TEST, action=31:0:0:c61f0d23-047e-464f-8fe6-cb76fa9a9b5a
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 9 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 10 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 11: start mysql-master-ip_start_0 on INACD-DB-SVR2-TEST (local)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=11:0:0:c61f0d23-047e-464f-8fe6-cb76fa9a9b5a op=mysql-master-ip_start_0 )
Feb  4 15:44:16 INACD-DB-SVR2-TEST lrmd: [10068]: info: rsc:mysql-master-ip:22: start
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 30 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 23 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 6 fired and confirmed
Feb  4 15:44:16 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/lrm (origin=local/crmd/54, version=0.84.6): ok (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=mysql-slave_monitor_0, magic=0:7;6:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522, cib=0.84.6) : Resource op removal
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: update_abort_priority: Abort action done superceeded by restart
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR1-TEST']/lrm": ok (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/transient_attributes (origin=local/crmd/55, version=0.84.7): ok (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=INACD-DB-SVR1-TEST, magic=NA, cib=0.84.7) : Transient attribute: removal
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR1-TEST']/transient_attributes": ok (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST IPaddr2[13937]: INFO: ip -f inet addr add 10.2.50.24/24 brd 10.2.50.255 dev eth0
Feb  4 15:44:16 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-master-ip:start:stderr) 2010/02/04_15:44:16 INFO: ip -f inet addr add 10.2.50.24/24 brd 10.2.50.255 dev eth0
Feb  4 15:44:16 INACD-DB-SVR2-TEST IPaddr2[13937]: INFO: ip link set eth0 up
Feb  4 15:44:16 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-master-ip:start:stderr) 2010/02/04_15:44:16 INFO: ip link set eth0 up
Feb  4 15:44:16 INACD-DB-SVR2-TEST IPaddr2[13937]: INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-10.2.50.24 eth0 10.2.50.24 auto not_used not_used
Feb  4 15:44:16 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-master-ip:start:stderr) 2010/02/04_15:44:16 INFO: /usr/lib64/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-10.2.50.24 eth0 10.2.50.24 auto not_used not_used
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-master-ip_start_0 (call=22, rc=0, cib-update=56, confirmed=true) ok
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master-ip_start_0 (11) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: run_graph: ====================================================
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: notice: run_graph: Transition 0 (Complete=14, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-error-2.bz2): Stopped
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_graph_trigger: Transition 0 is now complete
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 57: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke_callback: Invoking the PE: query=57, ref=pe_calc-dc-1265294656-19, seq=140, quorate=0
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR2-TEST is online
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master	(ocf::dk:mysql):	Stopped 
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave	(ocf::dk:mysql):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: clone_print:  Clone Set: cl-pingd
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Started: [ INACD-DB-SVR2-TEST ]
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Stopped: [ pingd:0 ]
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR2-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR1-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: info: native_merge_weights: mysql-master: Rolling back scores from mysql-master-ip
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: info: native_merge_weights: mysql-master: Rolling back scores from mysql-slave-ip
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: WARN: native_color: Resource mysql-master cannot run anywhere
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: WARN: native_color: Resource pingd:0 cannot run anywhere
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (10s) for mysql-master-ip on INACD-DB-SVR2-TEST
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master	(Stopped)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:0	(Stopped)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:1	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR2-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR1-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: unpack_graph: Unpacked transition 1: 1 actions in 1 synapses
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1265294656-19) derived from /var/lib/pengine/pe-warn-58.bz2
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 11: monitor mysql-master-ip_monitor_10000 on INACD-DB-SVR2-TEST (local)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_lrm_rsc_op: Performing key=11:1:0:c61f0d23-047e-464f-8fe6-cb76fa9a9b5a op=mysql-master-ip_monitor_10000 )
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: WARN: process_pe_message: Transition 1: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-58.bz2
Feb  4 15:44:16 INACD-DB-SVR2-TEST pengine: [10070]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: process_lrm_event: LRM operation mysql-master-ip_monitor_10000 (call=23, rc=0, cib-update=58, confirmed=false) ok
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master-ip_monitor_10000 (11) confirmed on INACD-DB-SVR2-TEST (rc=0)
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: run_graph: ====================================================
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: notice: run_graph: Transition 1 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-58.bz2): Complete
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: te_graph_trigger: Transition 1 is now complete
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: notify_crmd: Transition 1 status: done - <null>
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  4 15:44:16 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: Starting PEngine Recheck Timer
Feb  4 15:44:18 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Up, 100 Mbps full duplex, receive & transmit flow control ON
Feb  4 15:44:20 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Down
Feb  4 15:44:23 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Feb  4 15:44:57 INACD-DB-SVR2-TEST crmd: [10071]: WARN: cib_rsc_callback: Resource update 26 failed: (rc=-41) Remote node did not respond
Feb  4 15:45:06 INACD-DB-SVR2-TEST mysql[14404]: INFO: MySQL monitor succeded
Feb  4 15:45:06 INACD-DB-SVR2-TEST lrmd: [10068]: info: RA output: (mysql-slave:monitor:stderr) 2010/02/04_15:45:06 INFO: MySQL monitor succeded
Feb  4 15:45:33 INACD-DB-SVR2-TEST ntpd[3232]: Listening on interface #7 eth0, 10.2.50.24#123 Enabled
Feb  4 15:45:33 INACD-DB-SVR2-TEST ntpd[3232]: new interface(s) found: waking up resolver
Feb  4 15:45:57 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Down
Feb  4 15:45:59 INACD-DB-SVR2-TEST kernel: bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Feb  4 15:47:17 INACD-DB-SVR2-TEST sshd[15333]: Accepted publickey for root from 172.16.0.20 port 60364 ssh2
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Transitional membership event on ring 144: memb=1, new=0, lost=0
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: memb: INACD-DB-SVR2-TEST 38
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] notice: pcmk_peer_update: Stable membership event on ring 144: memb=2, new=1, lost=0
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: Node 36/INACD-DB-SVR1-TEST is now: member
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: NEW:  INACD-DB-SVR1-TEST 36
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: MEMB: INACD-DB-SVR1-TEST 36
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: pcmk_peer_update: MEMB: INACD-DB-SVR2-TEST 38
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: send_member_notification: Sending membership update 144 to 2 children
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: notice: ais_dispatch: Membership 144: quorum acquired
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: crm_update_peer: Node INACD-DB-SVR1-TEST: id=36 state=member (new) addr=r(0) ip(10.250.250.36)  votes=1 born=136 seen=144 proc=00000000000000000000000000013312
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: notice: ais_dispatch: Membership 144: quorum acquired
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_status_callback: status: INACD-DB-SVR1-TEST is now member (was lost)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_peer: Node INACD-DB-SVR1-TEST: id=36 state=member (new) addr=r(0) ip(10.250.250.36)  votes=1 born=136 seen=144 proc=00000000000000000000000000013312
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_quorum: Updating quorum status to true (call=63)
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: update_member: 0x676c00 Node 36 (INACD-DB-SVR1-TEST) born on: 144
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [pcmk  ] info: send_member_notification: Sending membership update 144 to 2 children
Feb  4 15:47:28 INACD-DB-SVR2-TEST corosync[10060]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/lrm (origin=local/crmd/59, version=0.84.9): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/transient_attributes (origin=local/crmd/60, version=0.84.9): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/61, version=0.84.9): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: log_data_element: cib:diff: - <cib have-quorum="0" admin_epoch="0" epoch="84" num_updates="10" />
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: log_data_element: cib:diff: + <cib have-quorum="1" admin_epoch="0" epoch="85" num_updates="1" />
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/63, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: ais_dispatch: Membership 144: quorum retained
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=INACD-DB-SVR1-TEST/INACD-DB-SVR1-TEST/(null), version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR1-TEST']/lrm": ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR1-TEST']/transient_attributes": ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: need_abort: Aborting on change to have-quorum
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: Membership changed: 140 -> 144 - join restart
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 66: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=do_state_transition ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/65, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: update_dc: Unset DC INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: join_make_offer: Making join offers based on membership 144
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_offer_all: A new node joined the cluster
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_offer_all: join-3: Waiting on 2 outstanding join acks
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [15457]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-84.raw
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: ais_dispatch: Membership 144: quorum retained
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [15457]: info: write_cib_contents: Wrote version 0.85.0 of the CIB to disk (digest: 9ca7e8994ccf0ae0058d7ab1be3bd236)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [15457]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.XA5c9E (digest: /var/lib/heartbeat/crm/cib.NlE8Wk)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/67, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: update_dc: Set DC to INACD-DB-SVR2-TEST (3.0.1)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/70, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 2 cluster nodes responded to the join offer.
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_finalize: join-3: Syncing the CIB from INACD-DB-SVR2-TEST to the rest of the cluster
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/73, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/74, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST lrmd: [10068]: debug: stonithRA plugin: provider attribute is not needed and will be ignored.
Feb  4 15:47:28 INACD-DB-SVR2-TEST lrmd: [10068]: WARN: stonithRA plugin: no short description in apcmastersnmp's metadata.
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/75, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/transient_attributes (origin=INACD-DB-SVR1-TEST/crmd/7, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/lrm (origin=INACD-DB-SVR1-TEST/crmd/8, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_ack: join-3: Updating node state to member for INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_ack: join-3: Updating node state to member for INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR1-TEST']/lrm (origin=local/crmd/76, version=0.85.1): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR1-TEST']/lrm": ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: crm_update_quorum: Updating quorum status to true (call=82)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-mysql-master (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 83: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='INACD-DB-SVR2-TEST']/lrm (origin=local/crmd/78, version=0.85.3): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:267 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=pingd:1_monitor_0, magic=0:7;16:0:7:a0caa6eb-4af2-4de0-8286-92abff7d2522, cib=0.85.3) : Resource op removal
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: erase_xpath_callback: Deletion of "//node_state[@uname='INACD-DB-SVR2-TEST']/lrm": ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 84: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_update_diff: Detected LRM refresh - 7 resources updated: Skipping all resource events
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:227 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.85.4) : LRM Refresh
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 85: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/80, version=0.85.4): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST cib: [10067]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/82, version=0.85.4): ok (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-mysql-master (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke_callback: Invoking the PE: query=85, ref=pe_calc-dc-1265294848-31, seq=144, quorate=1
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-mysql-master-ip (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-mysql-master-ip (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: pingd (3)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-pingd:0 (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST attrd: [10069]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-pingd:0 (<null>)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR1-TEST is online
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR2-TEST is online
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master	(ocf::dk:mysql):	Stopped 
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave	(ocf::dk:mysql):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: clone_print:  Clone Set: cl-pingd
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Started: [ INACD-DB-SVR2-TEST ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Stopped: [ pingd:0 ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR2-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR1-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (10s) for mysql-master on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (180s) for mysql-master on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (30s) for pingd:0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Start mysql-master	(INACD-DB-SVR1-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Start pingd:0	(INACD-DB-SVR1-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:1	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR2-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR1-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: unpack_graph: Unpacked transition 2: 16 actions in 16 synapses
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1265294848-31) derived from /var/lib/pengine/pe-input-136.bz2
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 10: monitor mysql-master_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 11: monitor mysql-master-ip_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 12: monitor mysql-slave_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 13: monitor mysql-slave-ip_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 14: monitor pingd:0_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 15: monitor apcstonith-INACD-DB-SVR2-TEST_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 16: monitor apcstonith-INACD-DB-SVR1-TEST_monitor_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:28 INACD-DB-SVR2-TEST pengine: [10070]: info: process_pe_message: Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-136.bz2
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master_monitor_0 (10) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-slave_monitor_0 (12) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master-ip_monitor_0 (11) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:28 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-slave-ip_monitor_0 (13) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action apcstonith-INACD-DB-SVR1-TEST_monitor_0 (16) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action apcstonith-INACD-DB-SVR2-TEST_monitor_0 (15) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action pingd:0_monitor_0 (14) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 9: probe_complete probe_complete on INACD-DB-SVR1-TEST - no waiting
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 8 fired and confirmed
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 18: start mysql-master_start_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 31 fired and confirmed
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 27: start pingd:0_start_0 on INACD-DB-SVR1-TEST
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=INACD-DB-SVR1-TEST, magic=NA, cib=0.85.12) : Transient attribute: update
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: update_abort_priority: Abort action done superceeded by restart
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action pingd:0_start_0 (27) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:29 INACD-DB-SVR2-TEST crmd: [10071]: info: te_pseudo_action: Pseudo action 32 fired and confirmed
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master_start_0 (18) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: run_graph: ====================================================
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: notice: run_graph: Transition 2 (Complete=13, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pengine/pe-input-136.bz2): Stopped
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: te_graph_trigger: Transition 2 is now complete
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 86: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke_callback: Invoking the PE: query=86, ref=pe_calc-dc-1265294855-42, seq=144, quorate=1
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR1-TEST is online
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR2-TEST is online
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master	(ocf::dk:mysql):	Started INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave	(ocf::dk:mysql):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: clone_print:  Clone Set: cl-pingd
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Started: [ INACD-DB-SVR1-TEST INACD-DB-SVR2-TEST ]
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR2-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR1-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (10s) for mysql-master on INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (180s) for mysql-master on INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: RecurringOp:  Start recurring monitor (30s) for pingd:0 on INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master	(Started INACD-DB-SVR1-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:0	(Started INACD-DB-SVR1-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:1	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR2-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR1-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: unpack_graph: Unpacked transition 3: 3 actions in 3 synapses
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1265294855-42) derived from /var/lib/pengine/pe-input-137.bz2
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 13: monitor mysql-master_monitor_10000 on INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 14: monitor mysql-master_monitor_180000 on INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: te_rsc_command: Initiating action 23: monitor pingd:0_monitor_30000 on INACD-DB-SVR1-TEST
Feb  4 15:47:35 INACD-DB-SVR2-TEST pengine: [10070]: info: process_pe_message: Transition 3: PEngine Input stored in: /var/lib/pengine/pe-input-137.bz2
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action pingd:0_monitor_30000 (23) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:35 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master_monitor_10000 (13) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: info: match_graph_event: Action mysql-master_monitor_180000 (14) confirmed on INACD-DB-SVR1-TEST (rc=0)
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: info: run_graph: ====================================================
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: notice: run_graph: Transition 3 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-137.bz2): Complete
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: info: te_graph_trigger: Transition 3 is now complete
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: info: notify_crmd: Transition 3 status: done - <null>
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  4 15:47:36 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: Starting PEngine Recheck Timer
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=INACD-DB-SVR1-TEST, magic=NA, cib=0.85.18) : Transient attribute: update
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke: Query 87: Requesting the current CIB: S_POLICY_ENGINE
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_pe_invoke_callback: Invoking the PE: query=87, ref=pe_calc-dc-1265294859-46, seq=144, quorate=1
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR1-TEST is online
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: info: determine_online_status: Node INACD-DB-SVR2-TEST is online
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master	(ocf::dk:mysql):	Started INACD-DB-SVR1-TEST
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-master-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave	(ocf::dk:mysql):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: mysql-slave-ip	(ocf::dk:IPaddr2):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: clone_print:  Clone Set: cl-pingd
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: short_print:      Started: [ INACD-DB-SVR1-TEST INACD-DB-SVR2-TEST ]
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR2-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: native_print: apcstonith-INACD-DB-SVR1-TEST	(stonith:apcmastersnmp):	Started INACD-DB-SVR2-TEST
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master	(Started INACD-DB-SVR1-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-master-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource mysql-slave-ip	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:0	(Started INACD-DB-SVR1-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource pingd:1	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR2-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: notice: LogActions: Leave resource apcstonith-INACD-DB-SVR1-TEST	(Started INACD-DB-SVR2-TEST)
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: unpack_graph: Unpacked transition 4: 0 actions in 0 synapses
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1265294859-46) derived from /var/lib/pengine/pe-input-138.bz2
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: run_graph: ====================================================
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: notice: run_graph: Transition 4 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-138.bz2): Complete
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: te_graph_trigger: Transition 4 is now complete
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: notify_crmd: Transition 4 status: done - <null>
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  4 15:47:39 INACD-DB-SVR2-TEST crmd: [10071]: info: do_state_transition: Starting PEngine Recheck Timer
Feb  4 15:47:39 INACD-DB-SVR2-TEST pengine: [10070]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/pengine/pe-input-138.bz2
Feb  4 15:47:47 INACD-DB-SVR2-TEST sshd[15605]: Accepted publickey for root from 10.2.50.36 port 48373 ssh2
Feb  4 15:47:47 INACD-DB-SVR2-TEST sshd[15635]: Accepted publickey for root from 10.2.50.36 port 48374 ssh2
Feb  4 15:47:47 INACD-DB-SVR2-TEST sshd[15667]: Accepted publickey for root from 10.2.50.36 port 48375 ssh2
Feb  4 15:47:48 INACD-DB-SVR2-TEST logger: Mark:HB_REPORT:1265294866
Feb  4 15:47:48 INACD-DB-SVR2-TEST sshd[15724]: Accepted publickey for root from 10.2.50.36 port 48376 ssh2
