Apr 27 05:31:00 s1 crm_attribute: [27670]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s1']//lrm_resource[@id='redis-serv-gr:0'] (origin=local/crmd/170, version=0.354.46): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: delete_resource: Removing resource redis-serv-gr:0 for 27689_crm_resource (internal) on s1
Apr 27 05:31:04 s1 crmd: [6959]: info: lrm_remove_deleted_op: Removing op redis-serv-gr:0_monitor_5000:34 for deleted resource redis-serv-gr:0
Apr 27 05:31:04 s1 crmd: [6959]: info: notify_deleted: Notifying 27689_crm_resource on s1 that redis-serv-gr:0 was deleted
Apr 27 05:31:04 s1 crmd: [6959]: WARN: decode_transition_key: Bad UUID (crm-resource-27689) in sscanf result (3) for 0:0:crm-resource-27689
Apr 27 05:31:04 s1 crmd: [6959]: info: send_direct_ack: ACK'ing resource op redis-serv-gr:0_delete_60000 from 0:0:crm-resource-27689: lrm_invoke-lrmd-1335533464-164
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s1']//lrm_resource[@id='redis-serv-gr:0'] (origin=local/crmd/171, version=0.354.47): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:291 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=redis-serv-gr:0_last_0, magic=0:0;12:61:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.354.47) : Resource op removal
Apr 27 05:31:04 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:291 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=redis-serv-gr:0_last_0, magic=0:0;12:61:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.354.47) : Resource op removal
Apr 27 05:31:04 s1 crmd: [6959]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Apr 27 05:31:04 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke: Query 174: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=s2/s2/(null), version=0.354.47): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke: Query 175: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: - <cib admin_epoch="0" epoch="354" num_updates="47" >
Apr 27 05:31:04 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:124 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.355.1) : Non-status change
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -   <configuration >
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke: Query 176: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -     <crm_config >
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -       <cluster_property_set id="cib-bootstrap-options" >
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -         <nvpair value="1335532447" id="cib-bootstrap-options-last-lrm-refresh" />
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -       </cluster_property_set>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -     </crm_config>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: -   </configuration>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: - </cib>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: + <cib validate-with="pacemaker-1.0" crm_feature_set="3.0.5" have-quorum="1" admin_epoch="0" epoch="355" num_updates="1" cib-last-written="Fri Apr 27 05:15:42 2012" dc-uuid="s1" >
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +   <configuration >
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +     <crm_config >
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +       <cluster_property_set id="cib-bootstrap-options" >
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +         <nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1335533464" />
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +       </cluster_property_set>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +     </crm_config>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: +   </configuration>
Apr 27 05:31:04 s1 cib: [6955]: info: cib:diff: + </cib>
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/173, version=0.355.1): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=176, ref=pe_calc-dc-1335533464-165, seq=364, quorate=1
Apr 27 05:31:04 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:04 s1 crmd: [6959]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Apr 27 05:31:04 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:31:04 s1 crmd: [6959]: info: config_query_callback: Checking for expired actions every 900000ms
Apr 27 05:31:04 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:31:04 s1 crmd: [6959]: info: config_query_callback: Sending expected-votes=2 to corosync
Apr 27 05:31:04 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:31:04 s1 crmd: [6959]: info: ais_dispatch_message: Membership 364: quorum retained
Apr 27 05:31:04 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:04 s1 crmd: [6959]: info: crmd_ais_dispatch: Setting expected votes to 2
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/179, version=0.355.7): ok (rc=0)
Apr 27 05:31:04 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:04 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:04 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:04 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:04 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:04 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:04 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:04 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:04 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s1 -> s2)
Apr 27 05:31:04 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:0#011(s1)
Apr 27 05:31:04 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:04 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-redis-serv-gr:0 (<null>)
Apr 27 05:31:04 s1 attrd: [6957]: notice: attrd_perform_update: Sent delete 96: node=s1, attr=fail-count-redis-serv-gr:0, id=<n/a>, set=(null), section=status
Apr 27 05:31:04 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:178 - Triggered transition abort (complete=1, tag=transient_attributes, id=s1, magic=NA, cib=0.355.8) : Transient attribute: removal
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke: Query 180: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:04 s1 attrd: [6957]: notice: attrd_perform_update: Sent delete -22: node=s1, attr=fail-count-redis-serv-gr:0, id=<n/a>, set=(null), section=status
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=180, ref=pe_calc-dc-1335533464-166, seq=364, quorate=1
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s2']//lrm_resource[@id='redis-serv-gr:0'] (origin=s2/crmd/48, version=0.355.8): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:291 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=redis-serv-gr:0_last_0, magic=0:7;5:47:7:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.355.9) : Resource op removal
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke: Query 181: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=181, ref=pe_calc-dc-1335533464-167, seq=364, quorate=1
Apr 27 05:31:04 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533464-165 is obsolete
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=s2/s2/(null), version=0.355.8): ok (rc=0)
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s2']//lrm_resource[@id='redis-serv-gr:0'] (origin=s2/crmd/49, version=0.355.9): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:291 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=redis-serv-gr:0_last_0, magic=0:7;5:47:7:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.355.9) : Resource op removal
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke: Query 182: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:04 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=s2/crmd/51, version=0.355.10): ok (rc=0)
Apr 27 05:31:04 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=182, ref=pe_calc-dc-1335533464-168, seq=364, quorate=1
Apr 27 05:31:05 s1 pengine: [6958]: notice: process_pe_message: Transition 64: PEngine Input stored in: /var/lib/pengine/pe-input-1619.bz2
Apr 27 05:31:05 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s1 -> s2)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:0#011(s1)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:05 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533464-166 is obsolete
Apr 27 05:31:05 s1 pengine: [6958]: notice: process_pe_message: Transition 65: PEngine Input stored in: /var/lib/pengine/pe-input-1620.bz2
Apr 27 05:31:05 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s1 -> s2)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:0#011(s1)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:05 s1 pengine: [6958]: notice: process_pe_message: Transition 66: PEngine Input stored in: /var/lib/pengine/pe-input-1620.bz2
Apr 27 05:31:05 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533464-167 is obsolete
Apr 27 05:31:05 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s1 -> s2)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:0#011(s1)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:05 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:05 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 67: 16 actions in 16 synapses
Apr 27 05:31:05 s1 crmd: [6959]: info: do_te_invoke: Processing graph 67 (ref=pe_calc-dc-1335533464-168) derived from /var/lib/pengine/pe-input-1621.bz2
Apr 27 05:31:05 s1 crmd: [6959]: info: te_rsc_command: Initiating action 6: monitor redis-serv-gr:0_monitor_0 on s1 (local)
Apr 27 05:31:05 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=6:67:7:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_monitor_0 )
Apr 27 05:31:05 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 probe[36] (pid 27691)
Apr 27 05:31:05 s1 crmd: [6959]: info: te_rsc_command: Initiating action 2: cancel redis-serv-gr:1_monitor_10000 on s2
Apr 27 05:31:05 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Apr 27 05:31:05 s1 crmd: [6959]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1335533465-11 from s2
Apr 27 05:31:05 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_10000 (2) confirmed on s2 (rc=0)
Apr 27 05:31:05 s1 pengine: [6958]: notice: process_pe_message: Transition 67: PEngine Input stored in: /var/lib/pengine/pe-input-1621.bz2
Apr 27 05:31:05 s1 crm_attribute: [27743]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:31:05 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-redis-serv-gr:0 (5)
Apr 27 05:31:05 s1 attrd: [6957]: notice: attrd_perform_update: Sent update 104: master-redis-serv-gr:0=5
Apr 27 05:31:05 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-s1-master-redis-serv-gr.0, name=master-redis-serv-gr:0, value=5, magic=NA, cib=0.355.17) : Transient attribute: update
Apr 27 05:31:05 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Apr 27 05:31:05 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:31:05 s1 lrmd: [6956]: info: operation monitor[36] on redis-serv-gr:0 for client 6959: pid 27691 exited with return code 0
Apr 27 05:31:05 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_monitor_0 (call=36, rc=0, cib-update=183, confirmed=true) ok
Apr 27 05:31:05 s1 crmd: [6959]: WARN: status_from_rc: Action 6 (redis-serv-gr:0_monitor_0) on s1 failed (target: 7 vs. rc: 0): Error
Apr 27 05:31:05 s1 crmd: [6959]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=redis-serv-gr:0_last_failure_0, magic=0:0;6:67:7:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.355.18) : Event failed
Apr 27 05:31:05 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_monitor_0 (6) confirmed on s1 (rc=4)
Apr 27 05:31:05 s1 crmd: [6959]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on s1 (local) - no waiting
Apr 27 05:31:05 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:05 s1 crmd: [6959]: notice: run_graph: Transition 67 (Complete=4, Pending=0, Fired=0, Skipped=10, Incomplete=2, Source=/var/lib/pengine/pe-input-1621.bz2): Stopped
Apr 27 05:31:05 s1 crmd: [6959]: info: te_graph_trigger: Transition 67 is now complete
Apr 27 05:31:05 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:05 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:05 s1 crmd: [6959]: info: do_pe_invoke: Query 184: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:05 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=184, ref=pe_calc-dc-1335533465-172, seq=364, quorate=1
Apr 27 05:31:05 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:05 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:0_last_failure_0 found resource redis-serv-gr:0 active on s1
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:31:05 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:31:05 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Restart ip-redis-gr#011(Started s1)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:0#011(Slave -> Master s1)
Apr 27 05:31:05 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:1#011(Slave s2)
Apr 27 05:31:05 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:05 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 68: 9 actions in 9 synapses
Apr 27 05:31:05 s1 crmd: [6959]: info: do_te_invoke: Processing graph 68 (ref=pe_calc-dc-1335533465-172) derived from /var/lib/pengine/pe-input-1622.bz2
Apr 27 05:31:05 s1 crmd: [6959]: info: te_rsc_command: Initiating action 6: stop ip-redis-gr_stop_0 on s1 (local)
Apr 27 05:31:05 s1 lrmd: [6956]: info: cancel_op: operation monitor[35] on ip-redis-gr for client 6959, its parameters: CRM_meta_name=[monitor] cidr_netmask=[24] crm_feature_set=[3.0.5] CRM_meta_timeout=[30000] CRM_meta_interval=[10000] nic=[eth0] ip=[192.168.1.15]  cancelled
Apr 27 05:31:05 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=6:68:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=ip-redis-gr_stop_0 )
Apr 27 05:31:05 s1 lrmd: [6956]: info: rsc:ip-redis-gr stop[37] (pid 27744)
Apr 27 05:31:05 s1 crmd: [6959]: info: te_rsc_command: Initiating action 16: monitor redis-serv-gr:1_monitor_10000 on s2
Apr 27 05:31:05 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
Apr 27 05:31:05 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_monitor_10000 (call=35, status=1, cib-update=0, confirmed=true) Cancelled
Apr 27 05:31:05 s1 crmd: [6959]: info: te_rsc_command: Initiating action 11: promote redis-serv-gr:0_promote_0 on s1 (local)
Apr 27 05:31:05 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=11:68:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_promote_0 )
Apr 27 05:31:05 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 promote[38] (pid 27745)
Apr 27 05:31:05 s1 pengine: [6958]: notice: process_pe_message: Transition 68: PEngine Input stored in: /var/lib/pengine/pe-input-1622.bz2
Apr 27 05:31:05 s1 IPaddr2[27744]: INFO: IP status = ok, IP_CIP=
Apr 27 05:31:05 s1 lrmd: [6956]: info: operation stop[37] on ip-redis-gr for client 6959: pid 27744 exited with return code 0
Apr 27 05:31:05 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_stop_0 (call=37, rc=0, cib-update=185, confirmed=true) ok
Apr 27 05:31:05 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_stop_0 (6) confirmed on s1 (rc=0)
Apr 27 05:31:05 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 2 fired and confirmed
Apr 27 05:31:05 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_10000 (16) confirmed on s2 (rc=0)
Apr 27 05:31:06 s1 crmd: [6959]: notice: do_lrm_invoke: Not creating resource for a delete event: (null)
Apr 27 05:31:06 s1 crmd: [6959]: info: notify_deleted: Notifying 27839_crm_resource on s1 that redis-serv-gr:1 was deleted
Apr 27 05:31:06 s1 crmd: [6959]: WARN: decode_transition_key: Bad UUID (crm-resource-27839) in sscanf result (3) for 0:0:crm-resource-27839
Apr 27 05:31:06 s1 crmd: [6959]: info: send_direct_ack: ACK'ing resource op redis-serv-gr:1_delete_60000 from 0:0:crm-resource-27839: lrm_invoke-lrmd-1335533466-177
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s1']//lrm_resource[@id='redis-serv-gr:1'] (origin=local/crmd/186, version=0.355.21): ok (rc=0)
Apr 27 05:31:06 s1 crmd: [6959]: WARN: decode_transition_key: Bad UUID (crm-resource-27839) in sscanf result (3) for 0:0:crm-resource-27839
Apr 27 05:31:06 s1 crmd: [6959]: info: send_direct_ack: ACK'ing resource op redis-serv-gr:1_delete_60000 from 0:0:crm-resource-27839: lrm_invoke-lrmd-1335533466-178
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: - <cib admin_epoch="0" epoch="355" num_updates="21" >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -   <configuration >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -     <crm_config >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -       <cluster_property_set id="cib-bootstrap-options" >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -         <nvpair value="1335533464" id="cib-bootstrap-options-last-lrm-refresh" />
Apr 27 05:31:06 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:124 - Triggered transition abort (complete=0, tag=diff, id=(null), magic=NA, cib=0.356.1) : Non-status change
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -       </cluster_property_set>
Apr 27 05:31:06 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -     </crm_config>
Apr 27 05:31:06 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: -   </configuration>
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: - </cib>
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: + <cib validate-with="pacemaker-1.0" crm_feature_set="3.0.5" have-quorum="1" admin_epoch="0" epoch="356" num_updates="1" cib-last-written="Fri Apr 27 05:31:04 2012" dc-uuid="s1" >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +   <configuration >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +     <crm_config >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +       <cluster_property_set id="cib-bootstrap-options" >
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +         <nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1335533466" />
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +       </cluster_property_set>
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +     </crm_config>
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: +   </configuration>
Apr 27 05:31:06 s1 cib: [6955]: info: cib:diff: + </cib>
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/188, version=0.356.1): ok (rc=0)
Apr 27 05:31:06 s1 crmd: [6959]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Apr 27 05:31:06 s1 crmd: [6959]: info: config_query_callback: Checking for expired actions every 900000ms
Apr 27 05:31:06 s1 crmd: [6959]: info: config_query_callback: Sending expected-votes=2 to corosync
Apr 27 05:31:06 s1 crmd: [6959]: info: ais_dispatch_message: Membership 364: quorum retained
Apr 27 05:31:06 s1 crmd: [6959]: info: crmd_ais_dispatch: Setting expected votes to 2
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/191, version=0.356.2): ok (rc=0)
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s2']//lrm_resource[@id='redis-serv-gr:1'] (origin=s2/crmd/55, version=0.356.2): ok (rc=0)
Apr 27 05:31:06 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:291 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=redis-serv-gr:1_last_0, magic=0:0;12:62:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.356.3) : Resource op removal
Apr 27 05:31:06 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:178 - Triggered transition abort (complete=0, tag=transient_attributes, id=s2, magic=NA, cib=0.356.3) : Transient attribute: removal
Apr 27 05:31:06 s1 crmd: [6959]: ERROR: send_msg_via_ipc: Unknown Sub-system (27839_crm_resource)... discarding message.
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=s2/s2/(null), version=0.356.4): ok (rc=0)
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='s2']//lrm_resource[@id='redis-serv-gr:1'] (origin=s2/crmd/56, version=0.356.5): ok (rc=0)
Apr 27 05:31:06 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:291 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=redis-serv-gr:1_last_0, magic=0:0;12:62:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.356.5) : Resource op removal
Apr 27 05:31:06 s1 cib: [6955]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=s2/crmd/58, version=0.356.6): ok (rc=0)
Apr 27 05:31:06 s1 crm_attribute: [27884]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:31:06 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-redis-serv-gr:0 (10)
Apr 27 05:31:06 s1 attrd: [6957]: notice: attrd_perform_update: Sent update 111: master-redis-serv-gr:0=10
Apr 27 05:31:06 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-s1-master-redis-serv-gr.0, name=master-redis-serv-gr:0, value=10, magic=NA, cib=0.356.11) : Transient attribute: update
Apr 27 05:31:06 s1 lrmd: [6956]: info: operation promote[38] on redis-serv-gr:0 for client 6959: pid 27745 exited with return code 0
Apr 27 05:31:06 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_promote_0 (call=38, rc=0, cib-update=192, confirmed=true) ok
Apr 27 05:31:06 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_promote_0 (11) confirmed on s1 (rc=0)
Apr 27 05:31:06 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 22 fired and confirmed
Apr 27 05:31:06 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:06 s1 crmd: [6959]: notice: run_graph: Transition 68 (Complete=6, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pengine/pe-input-1622.bz2): Stopped
Apr 27 05:31:06 s1 crmd: [6959]: info: te_graph_trigger: Transition 68 is now complete
Apr 27 05:31:06 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:06 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:06 s1 crmd: [6959]: info: do_pe_invoke: Query 193: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:06 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=193, ref=pe_calc-dc-1335533466-179, seq=364, quorate=1
Apr 27 05:31:06 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:06 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:0_last_failure_0 found resource redis-serv-gr:0 active on s1
Apr 27 05:31:06 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:31:06 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:31:06 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:31:06 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:31:06 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:31:06 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s1)
Apr 27 05:31:06 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:0#011(Master s1)
Apr 27 05:31:06 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:1#011(s2)
Apr 27 05:31:06 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:06 s1 pengine: [6958]: notice: process_pe_message: Transition 69: PEngine Input stored in: /var/lib/pengine/pe-input-1623.bz2
Apr 27 05:31:06 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 69: 10 actions in 10 synapses
Apr 27 05:31:06 s1 crmd: [6959]: info: do_te_invoke: Processing graph 69 (ref=pe_calc-dc-1335533466-179) derived from /var/lib/pengine/pe-input-1623.bz2
Apr 27 05:31:06 s1 crmd: [6959]: info: te_rsc_command: Initiating action 6: start ip-redis-gr_start_0 on s1 (local)
Apr 27 05:31:06 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=6:69:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=ip-redis-gr_start_0 )
Apr 27 05:31:06 s1 lrmd: [6956]: info: rsc:ip-redis-gr start[39] (pid 27885)
Apr 27 05:31:06 s1 crmd: [6959]: info: te_rsc_command: Initiating action 12: monitor redis-serv-gr:0_monitor_5000 on s1 (local)
Apr 27 05:31:06 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=12:69:8:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_monitor_5000 )
Apr 27 05:31:06 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 monitor[40] (pid 27886)
Apr 27 05:31:06 s1 crmd: [6959]: info: te_rsc_command: Initiating action 5: monitor redis-serv-gr:1_monitor_0 on s2
Apr 27 05:31:06 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Apr 27 05:31:07 s1 crmd: [6959]: WARN: status_from_rc: Action 5 (redis-serv-gr:1_monitor_0) on s2 failed (target: 7 vs. rc: 0): Error
Apr 27 05:31:07 s1 crmd: [6959]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=redis-serv-gr:1_last_failure_0, magic=0:0;5:69:7:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.356.13) : Event failed
Apr 27 05:31:07 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Apr 27 05:31:07 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:31:07 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_0 (5) confirmed on s2 (rc=4)
Apr 27 05:31:07 s1 crmd: [6959]: info: te_rsc_command: Initiating action 4: probe_complete probe_complete on s2 - no waiting
Apr 27 05:31:07 s1 IPaddr2[27885]: INFO: ip -f inet addr add 192.168.1.15/24 brd 192.168.1.255 dev eth0
Apr 27 05:31:07 s1 IPaddr2[27885]: INFO: ip link set eth0 up
Apr 27 05:31:07 s1 IPaddr2[27885]: INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.1.15 eth0 192.168.1.15 auto not_used not_used
Apr 27 05:31:07 s1 lrmd: [6956]: info: operation start[39] on ip-redis-gr for client 6959: pid 27885 exited with return code 0
Apr 27 05:31:07 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_start_0 (call=39, rc=0, cib-update=194, confirmed=true) ok
Apr 27 05:31:07 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_start_0 (6) confirmed on s1 (rc=0)
Apr 27 05:31:07 s1 crm_attribute: [27988]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:31:07 s1 lrmd: [6956]: info: operation monitor[40] on redis-serv-gr:0 for client 6959: pid 27886 exited with return code 8
Apr 27 05:31:07 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_monitor_5000 (call=40, rc=8, cib-update=195, confirmed=false) master
Apr 27 05:31:07 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_monitor_5000 (12) confirmed on s1 (rc=0)
Apr 27 05:31:07 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:07 s1 crmd: [6959]: notice: run_graph: Transition 69 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=1, Source=/var/lib/pengine/pe-input-1623.bz2): Stopped
Apr 27 05:31:07 s1 crmd: [6959]: info: te_graph_trigger: Transition 69 is now complete
Apr 27 05:31:07 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:07 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:07 s1 crmd: [6959]: info: do_pe_invoke: Query 196: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:07 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=196, ref=pe_calc-dc-1335533467-184, seq=364, quorate=1
Apr 27 05:31:07 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:07 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:0_last_failure_0 found resource redis-serv-gr:0 active on s1
Apr 27 05:31:07 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:31:07 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:31:07 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:31:07 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:31:07 s1 pengine: [6958]: notice: LogActions: Leave   ip-redis-gr#011(Started s1)
Apr 27 05:31:07 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:0#011(Master s1)
Apr 27 05:31:07 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:1#011(Slave s2)
Apr 27 05:31:07 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:07 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 70: 2 actions in 2 synapses
Apr 27 05:31:07 s1 crmd: [6959]: info: do_te_invoke: Processing graph 70 (ref=pe_calc-dc-1335533467-184) derived from /var/lib/pengine/pe-input-1624.bz2
Apr 27 05:31:07 s1 crmd: [6959]: info: te_rsc_command: Initiating action 8: monitor ip-redis-gr_monitor_10000 on s1 (local)
Apr 27 05:31:07 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=8:70:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=ip-redis-gr_monitor_10000 )
Apr 27 05:31:07 s1 lrmd: [6956]: info: rsc:ip-redis-gr monitor[41] (pid 27989)
Apr 27 05:31:07 s1 crmd: [6959]: info: te_rsc_command: Initiating action 16: monitor redis-serv-gr:1_monitor_10000 on s2
Apr 27 05:31:07 s1 pengine: [6958]: notice: process_pe_message: Transition 70: PEngine Input stored in: /var/lib/pengine/pe-input-1624.bz2
Apr 27 05:31:07 s1 lrmd: [6956]: info: operation monitor[41] on ip-redis-gr for client 6959: pid 27989 exited with return code 0
Apr 27 05:31:07 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_monitor_10000 (call=41, rc=0, cib-update=197, confirmed=false) ok
Apr 27 05:31:07 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_monitor_10000 (8) confirmed on s1 (rc=0)
Apr 27 05:31:07 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_10000 (16) confirmed on s2 (rc=0)
Apr 27 05:31:07 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:07 s1 crmd: [6959]: notice: run_graph: Transition 70 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1624.bz2): Complete
Apr 27 05:31:07 s1 crmd: [6959]: info: te_graph_trigger: Transition 70 is now complete
Apr 27 05:31:07 s1 crmd: [6959]: info: notify_crmd: Transition 70 status: done - <null>
Apr 27 05:31:07 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:07 s1 crmd: [6959]: info: do_state_transition: Starting PEngine Recheck Timer
Apr 27 05:31:12 s1 crm_attribute: [28070]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:31:18 s1 crm_attribute: [28152]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:31:23 s1 crm_attribute: [28237]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -D 
Apr 27 05:31:23 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-redis-serv-gr:0 (<null>)
Apr 27 05:31:23 s1 attrd: [6957]: notice: attrd_perform_update: Sent delete 113: node=s1, attr=master-redis-serv-gr:0, id=<n/a>, set=(null), section=status
Apr 27 05:31:23 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:178 - Triggered transition abort (complete=1, tag=transient_attributes, id=s1, magic=NA, cib=0.356.18) : Transient attribute: removal
Apr 27 05:31:23 s1 crmd: [6959]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Apr 27 05:31:23 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:23 s1 crmd: [6959]: info: do_pe_invoke: Query 198: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:23 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=198, ref=pe_calc-dc-1335533483-187, seq=364, quorate=1
Apr 27 05:31:23 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:23 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:0_last_failure_0 found resource redis-serv-gr:0 active on s1
Apr 27 05:31:23 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:31:23 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:23 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:23 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:23 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:23 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:23 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s1 -> s2)
Apr 27 05:31:23 s1 pengine: [6958]: notice: LogActions: Demote  redis-serv-gr:0#011(Master -> Slave s1)
Apr 27 05:31:23 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:23 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:23 s1 pengine: [6958]: notice: process_pe_message: Transition 71: PEngine Input stored in: /var/lib/pengine/pe-input-1625.bz2
Apr 27 05:31:23 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 71: 14 actions in 14 synapses
Apr 27 05:31:23 s1 crmd: [6959]: info: do_te_invoke: Processing graph 71 (ref=pe_calc-dc-1335533483-187) derived from /var/lib/pengine/pe-input-1625.bz2
Apr 27 05:31:23 s1 crmd: [6959]: info: te_rsc_command: Initiating action 8: stop ip-redis-gr_stop_0 on s1 (local)
Apr 27 05:31:23 s1 lrmd: [6956]: info: cancel_op: operation monitor[41] on ip-redis-gr for client 6959, its parameters: CRM_meta_name=[monitor] cidr_netmask=[24] crm_feature_set=[3.0.5] CRM_meta_timeout=[30000] CRM_meta_interval=[10000] nic=[eth0] ip=[192.168.1.15]  cancelled
Apr 27 05:31:23 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=8:71:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=ip-redis-gr_stop_0 )
Apr 27 05:31:23 s1 lrmd: [6956]: info: rsc:ip-redis-gr stop[42] (pid 28238)
Apr 27 05:31:23 s1 crmd: [6959]: info: te_rsc_command: Initiating action 2: cancel redis-serv-gr:0_monitor_5000 on s1 (local)
Apr 27 05:31:23 s1 lrmd: [6956]: info: cancel_op: operation monitor[40] with pid 28185 on redis-serv-gr:0 for client 6959, its parameters: CRM_meta_clone=[0] vservers=[s1test,s2test] CRM_meta_master_node_max=[1] CRM_meta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[false] CRM_meta_master_max=[1] CRM_meta_globally_unique=[false] crm_feature_set=[3.0.5] masterip=[192.168.1.15] CRM_meta_name=[monitor] CRM_meta_role=[Master] CRM_meta_interval=[5000] CRM_meta_timeout=[60000]  cancelled
Apr 27 05:31:23 s1 lrmd: [6956]: info: flush_op: process for operation monitor[40] with pid 28185 on redis-serv-gr:0 for client 6959 still running, flush delayed
Apr 27 05:31:23 s1 lrmd: [6956]: info: on_msg_cancel_op: operation 40 running, cancel pending
Apr 27 05:31:23 s1 crmd: [6959]: info: send_direct_ack: ACK'ing resource op redis-serv-gr:0_monitor_5000 from 2:71:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77: lrm_invoke-lrmd-1335533483-190
Apr 27 05:31:23 s1 crmd: [6959]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1335533483-190 from s1
Apr 27 05:31:23 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_monitor_5000 (2) confirmed on s1 (rc=0)
Apr 27 05:31:23 s1 crmd: [6959]: info: te_rsc_command: Initiating action 3: cancel redis-serv-gr:1_monitor_10000 on s2
Apr 27 05:31:23 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_monitor_10000 (call=41, status=1, cib-update=0, confirmed=true) Cancelled
Apr 27 05:31:23 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_monitor_5000 (call=40, status=1, cib-update=0, confirmed=true) Cancelled
Apr 27 05:31:23 s1 crmd: [6959]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1335533483-13 from s2
Apr 27 05:31:23 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_10000 (3) confirmed on s2 (rc=0)
Apr 27 05:31:23 s1 IPaddr2[28238]: INFO: IP status = ok, IP_CIP=
Apr 27 05:31:23 s1 lrmd: [6956]: info: operation stop[42] on ip-redis-gr for client 6959: pid 28238 exited with return code 0
Apr 27 05:31:23 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_stop_0 (call=42, rc=0, cib-update=200, confirmed=true) ok
Apr 27 05:31:23 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_stop_0 (8) confirmed on s1 (rc=0)
Apr 27 05:31:23 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
Apr 27 05:31:23 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 4 fired and confirmed
Apr 27 05:31:23 s1 crmd: [6959]: info: te_rsc_command: Initiating action 13: demote redis-serv-gr:0_demote_0 on s1 (local)
Apr 27 05:31:23 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=13:71:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_demote_0 )
Apr 27 05:31:23 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 demote[43] (pid 28273)
Apr 27 05:31:24 s1 lrmd: [6956]: info: operation demote[43] on redis-serv-gr:0 for client 6959: pid 28273 exited with return code 7
Apr 27 05:31:24 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_demote_0 (call=43, rc=7, cib-update=201, confirmed=true) not running
Apr 27 05:31:24 s1 crmd: [6959]: WARN: status_from_rc: Action 13 (redis-serv-gr:0_demote_0) on s1 failed (target: 0 vs. rc: 7): Error
Apr 27 05:31:24 s1 crmd: [6959]: WARN: update_failcount: Updating failcount for redis-serv-gr:0 on s1 after failed demote: rc=7 (update=value++, time=1335533484)
Apr 27 05:31:24 s1 crmd: [6959]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=redis-serv-gr:0_last_failure_0, magic=0:7;13:71:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.356.22) : Event failed
Apr 27 05:31:24 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Apr 27 05:31:24 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:31:24 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_demote_0 (13) confirmed on s1 (rc=4)
Apr 27 05:31:24 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Apr 27 05:31:24 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:24 s1 crmd: [6959]: notice: run_graph: Transition 71 (Complete=7, Pending=0, Fired=0, Skipped=6, Incomplete=1, Source=/var/lib/pengine/pe-input-1625.bz2): Stopped
Apr 27 05:31:24 s1 crmd: [6959]: info: te_graph_trigger: Transition 71 is now complete
Apr 27 05:31:24 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-redis-serv-gr:0 (1)
Apr 27 05:31:24 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:24 s1 attrd: [6957]: notice: attrd_perform_update: Sent update 117: fail-count-redis-serv-gr:0=1
Apr 27 05:31:24 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:24 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-redis-serv-gr:0 (1335533484)
Apr 27 05:31:24 s1 crmd: [6959]: info: do_pe_invoke: Query 202: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:24 s1 attrd: [6957]: notice: attrd_perform_update: Sent update 119: last-failure-redis-serv-gr:0=1335533484
Apr 27 05:31:24 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-s1-fail-count-redis-serv-gr.0, name=fail-count-redis-serv-gr:0, value=1, magic=NA, cib=0.356.23) : Transient attribute: update
Apr 27 05:31:24 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=202, ref=pe_calc-dc-1335533484-193, seq=364, quorate=1
Apr 27 05:31:24 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:24 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-s1-last-failure-redis-serv-gr.0, name=last-failure-redis-serv-gr:0, value=1335533484, magic=NA, cib=0.356.24) : Transient attribute: update
Apr 27 05:31:24 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:31:24 s1 crmd: [6959]: info: do_pe_invoke: Query 203: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:24 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:31:24 s1 crmd: [6959]: info: do_pe_invoke: Query 204: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:24 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:31:24 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=204, ref=pe_calc-dc-1335533484-194, seq=364, quorate=1
Apr 27 05:31:24 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:24 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:24 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s2)
Apr 27 05:31:24 s1 pengine: [6958]: notice: LogActions: Stop    redis-serv-gr:0#011(s1)
Apr 27 05:31:24 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:24 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533484-193 is obsolete
Apr 27 05:31:24 s1 pengine: [6958]: notice: process_pe_message: Transition 72: PEngine Input stored in: /var/lib/pengine/pe-input-1626.bz2
Apr 27 05:31:24 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:24 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:31:24 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:31:24 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:24 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:24 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:24 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s2)
Apr 27 05:31:24 s1 pengine: [6958]: notice: LogActions: Stop    redis-serv-gr:0#011(s1)
Apr 27 05:31:24 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:1#011(Slave -> Master s2)
Apr 27 05:31:24 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:24 s1 crmd: [6959]: WARN: destroy_action: Cancelling timer for action 2 (src=286)
Apr 27 05:31:24 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 73: 10 actions in 10 synapses
Apr 27 05:31:24 s1 crmd: [6959]: info: do_te_invoke: Processing graph 73 (ref=pe_calc-dc-1335533484-194) derived from /var/lib/pengine/pe-input-1627.bz2
Apr 27 05:31:24 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Apr 27 05:31:24 s1 crmd: [6959]: info: te_rsc_command: Initiating action 1: stop redis-serv-gr:0_stop_0 on s1 (local)
Apr 27 05:31:24 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=1:73:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_stop_0 )
Apr 27 05:31:24 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 stop[44] (pid 28313)
Apr 27 05:31:24 s1 pengine: [6958]: notice: process_pe_message: Transition 73: PEngine Input stored in: /var/lib/pengine/pe-input-1627.bz2
Apr 27 05:31:24 s1 lrmd: [6956]: info: RA output: (redis-serv-gr:0:stop:stderr) /usr/lib/ocf/resource.d//implix/redis6: 311: 
Apr 27 05:31:24 s1 lrmd: [6956]: info: RA output: (redis-serv-gr:0:stop:stderr) CRM_MASTER: not found
Apr 27 05:31:24 s1 lrmd: [6956]: info: RA output: (redis-serv-gr:0:stop:stderr) 
Apr 27 05:31:25 s1 lrmd: [6956]: info: operation stop[44] on redis-serv-gr:0 for client 6959: pid 28313 exited with return code 0
Apr 27 05:31:25 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_stop_0 (call=44, rc=0, cib-update=205, confirmed=true) ok
Apr 27 05:31:25 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_stop_0 (1) confirmed on s1 (rc=0)
Apr 27 05:31:25 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Apr 27 05:31:25 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 2 fired and confirmed
Apr 27 05:31:25 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Apr 27 05:31:25 s1 crmd: [6959]: info: te_rsc_command: Initiating action 11: promote redis-serv-gr:1_promote_0 on s2
Apr 27 05:31:26 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-s2-master-redis-serv-gr.1, name=master-redis-serv-gr:1, value=10, magic=NA, cib=0.356.26) : Transient attribute: update
Apr 27 05:31:26 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Apr 27 05:31:26 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:31:26 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_promote_0 (11) confirmed on s2 (rc=0)
Apr 27 05:31:26 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Apr 27 05:31:26 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:26 s1 crmd: [6959]: notice: run_graph: Transition 73 (Complete=7, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pengine/pe-input-1627.bz2): Stopped
Apr 27 05:31:26 s1 crmd: [6959]: info: te_graph_trigger: Transition 73 is now complete
Apr 27 05:31:26 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:26 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:26 s1 crmd: [6959]: info: do_pe_invoke: Query 206: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:26 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=206, ref=pe_calc-dc-1335533486-197, seq=364, quorate=1
Apr 27 05:31:26 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:26 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:31:26 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:31:26 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:31:26 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:26 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:26 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s2
Apr 27 05:31:26 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:26 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:26 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:26 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:1 on s2
Apr 27 05:31:26 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s2)
Apr 27 05:31:26 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:0#011(s1)
Apr 27 05:31:26 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:1#011(Master s2)
Apr 27 05:31:26 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:26 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 74: 7 actions in 7 synapses
Apr 27 05:31:26 s1 crmd: [6959]: info: do_te_invoke: Processing graph 74 (ref=pe_calc-dc-1335533486-197) derived from /var/lib/pengine/pe-input-1628.bz2
Apr 27 05:31:26 s1 pengine: [6958]: notice: process_pe_message: Transition 74: PEngine Input stored in: /var/lib/pengine/pe-input-1628.bz2
Apr 27 05:31:26 s1 crmd: [6959]: info: te_rsc_command: Initiating action 5: start ip-redis-gr_start_0 on s2
Apr 27 05:31:26 s1 crmd: [6959]: info: te_rsc_command: Initiating action 13: monitor redis-serv-gr:1_monitor_5000 on s2
Apr 27 05:31:26 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 14 fired and confirmed
Apr 27 05:31:26 s1 crmd: [6959]: info: te_rsc_command: Initiating action 7: start redis-serv-gr:0_start_0 on s1 (local)
Apr 27 05:31:26 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=7:74:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_start_0 )
Apr 27 05:31:26 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 start[45] (pid 28356)
Apr 27 05:31:26 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_start_0 (5) confirmed on s2 (rc=0)
Apr 27 05:31:26 s1 crmd: [6959]: info: te_rsc_command: Initiating action 6: monitor ip-redis-gr_monitor_10000 on s2
Apr 27 05:31:26 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_monitor_10000 (6) confirmed on s2 (rc=0)
Apr 27 05:31:26 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_5000 (13) confirmed on s2 (rc=0)
Apr 27 05:31:27 s1 crm_attribute: [28482]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:31:27 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-redis-serv-gr:0 (5)
Apr 27 05:31:27 s1 attrd: [6957]: notice: attrd_perform_update: Sent update 123: master-redis-serv-gr:0=5
Apr 27 05:31:27 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-s1-master-redis-serv-gr.0, name=master-redis-serv-gr:0, value=5, magic=NA, cib=0.356.31) : Transient attribute: update
Apr 27 05:31:27 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Apr 27 05:31:27 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:31:27 s1 lrmd: [6956]: info: operation start[45] on redis-serv-gr:0 for client 6959: pid 28356 exited with return code 0
Apr 27 05:31:27 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_start_0 (call=45, rc=0, cib-update=207, confirmed=true) ok
Apr 27 05:31:27 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_start_0 (7) confirmed on s1 (rc=0)
Apr 27 05:31:27 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Apr 27 05:31:27 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:27 s1 crmd: [6959]: notice: run_graph: Transition 74 (Complete=6, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-1628.bz2): Stopped
Apr 27 05:31:27 s1 crmd: [6959]: info: te_graph_trigger: Transition 74 is now complete
Apr 27 05:31:27 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:27 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:31:27 s1 crmd: [6959]: info: do_pe_invoke: Query 208: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:31:27 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=208, ref=pe_calc-dc-1335533487-202, seq=364, quorate=1
Apr 27 05:31:27 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:31:27 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:31:27 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:31:27 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:31:27 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:27 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:31:27 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:27 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:0 on s1
Apr 27 05:31:27 s1 pengine: [6958]: notice: LogActions: Leave   ip-redis-gr#011(Started s2)
Apr 27 05:31:27 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:0#011(Slave s1)
Apr 27 05:31:27 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:1#011(Master s2)
Apr 27 05:31:27 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:31:27 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 75: 1 actions in 1 synapses
Apr 27 05:31:27 s1 crmd: [6959]: info: do_te_invoke: Processing graph 75 (ref=pe_calc-dc-1335533487-202) derived from /var/lib/pengine/pe-input-1629.bz2
Apr 27 05:31:27 s1 crmd: [6959]: info: te_rsc_command: Initiating action 12: monitor redis-serv-gr:0_monitor_10000 on s1 (local)
Apr 27 05:31:27 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=12:75:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_monitor_10000 )
Apr 27 05:31:27 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 monitor[46] (pid 28487)
Apr 27 05:31:27 s1 pengine: [6958]: notice: process_pe_message: Transition 75: PEngine Input stored in: /var/lib/pengine/pe-input-1629.bz2
Apr 27 05:31:27 s1 crm_attribute: [28539]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:31:27 s1 lrmd: [6956]: info: operation monitor[46] on redis-serv-gr:0 for client 6959: pid 28487 exited with return code 0
Apr 27 05:31:27 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_monitor_10000 (call=46, rc=0, cib-update=209, confirmed=false) ok
Apr 27 05:31:27 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_monitor_10000 (12) confirmed on s1 (rc=0)
Apr 27 05:31:27 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:31:27 s1 crmd: [6959]: notice: run_graph: Transition 75 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1629.bz2): Complete
Apr 27 05:31:27 s1 crmd: [6959]: info: te_graph_trigger: Transition 75 is now complete
Apr 27 05:31:27 s1 crmd: [6959]: info: notify_crmd: Transition 75 status: done - <null>
Apr 27 05:31:27 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:31:27 s1 crmd: [6959]: info: do_state_transition: Starting PEngine Recheck Timer
Apr 27 05:31:37 s1 crm_attribute: [28592]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:31:48 s1 crm_attribute: [28645]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:31:59 s1 crm_attribute: [28698]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:32:09 s1 crm_attribute: [28751]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:32:19 s1 crm_attribute: [28804]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:32:30 s1 crm_attribute: [28857]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:32:40 s1 crm_attribute: [28910]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:32:50 s1 crm_attribute: [28963]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:33:01 s1 crm_attribute: [29016]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:33:11 s1 crm_attribute: [29069]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 5 
Apr 27 05:33:14 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:178 - Triggered transition abort (complete=1, tag=transient_attributes, id=s2, magic=NA, cib=0.356.34) : Transient attribute: removal
Apr 27 05:33:14 s1 crmd: [6959]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Apr 27 05:33:14 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke: Query 210: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=210, ref=pe_calc-dc-1335533594-204, seq=364, quorate=1
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:33:14 s1 crmd: [6959]: info: process_graph_event: Detected action redis-serv-gr:1_monitor_5000 from a different transition: 74 vs. 75
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 crmd: [6959]: info: abort_transition_graph: process_graph_event:485 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=redis-serv-gr:1_monitor_5000, magic=0:9;13:74:8:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.356.36) : Old event
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:14 s1 crmd: [6959]: WARN: update_failcount: Updating failcount for redis-serv-gr:1 on s2 after failed monitor: rc=9 (update=value++, time=1335533594)
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke: Query 211: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=211, ref=pe_calc-dc-1335533594-205, seq=364, quorate=1
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s2 -> s1)
Apr 27 05:33:14 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-s2-fail-count-redis-serv-gr.1, name=fail-count-redis-serv-gr:1, value=1, magic=NA, cib=0.356.37) : Transient attribute: update
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:0#011(Slave -> Master s1)
Apr 27 05:33:14 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-s2-last-failure-redis-serv-gr.1, name=last-failure-redis-serv-gr:1, value=1335533594, magic=NA, cib=0.356.38) : Transient attribute: update
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Demote  redis-serv-gr:1#011(Master -> Slave s2)
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke: Query 212: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:14 s1 pengine: [6958]: notice: process_pe_message: Transition 76: PEngine Input stored in: /var/lib/pengine/pe-input-1630.bz2
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke: Query 213: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=213, ref=pe_calc-dc-1335533594-206, seq=364, quorate=1
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:14 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533594-204 is obsolete
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s2 -> s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:0#011(Slave -> Master s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Demote  redis-serv-gr:1#011(Master -> Slave s2)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Recover redis-serv-gr:1#011(Master s2)
Apr 27 05:33:14 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533594-205 is obsolete
Apr 27 05:33:14 s1 pengine: [6958]: notice: process_pe_message: Transition 77: PEngine Input stored in: /var/lib/pengine/pe-input-1631.bz2
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_rsc_op: Operation redis-serv-gr:1_last_failure_0 found resource redis-serv-gr:1 active on s2
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s2 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s2 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Move    ip-redis-gr#011(Started s2 -> s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:0#011(Slave -> Master s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Demote  redis-serv-gr:1#011(Master -> Slave s2)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Recover redis-serv-gr:1#011(Master s2)
Apr 27 05:33:14 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:33:14 s1 pengine: [6958]: notice: process_pe_message: Transition 78: PEngine Input stored in: /var/lib/pengine/pe-input-1632.bz2
Apr 27 05:33:14 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 78: 19 actions in 19 synapses
Apr 27 05:33:14 s1 crmd: [6959]: info: do_te_invoke: Processing graph 78 (ref=pe_calc-dc-1335533594-206) derived from /var/lib/pengine/pe-input-1632.bz2
Apr 27 05:33:14 s1 crmd: [6959]: info: te_rsc_command: Initiating action 9: stop ip-redis-gr_stop_0 on s2
Apr 27 05:33:14 s1 crmd: [6959]: info: te_rsc_command: Initiating action 1: cancel redis-serv-gr:0_monitor_10000 on s1 (local)
Apr 27 05:33:14 s1 lrmd: [6956]: info: cancel_op: operation monitor[46] on redis-serv-gr:0 for client 6959, its parameters: CRM_meta_clone=[0] vservers=[s1test,s2test] CRM_meta_master_node_max=[1] CRM_meta_clone_node_max=[1] CRM_meta_clone_max=[2] CRM_meta_notify=[false] CRM_meta_master_max=[1] CRM_meta_globally_unique=[false] crm_feature_set=[3.0.5] masterip=[192.168.1.15] CRM_meta_name=[monitor] CRM_meta_role=[Slave] CRM_meta_interval=[10000] CRM_meta_timeout=[60000]  cancelled
Apr 27 05:33:14 s1 crmd: [6959]: info: send_direct_ack: ACK'ing resource op redis-serv-gr:0_monitor_10000 from 1:78:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77: lrm_invoke-lrmd-1335533594-209
Apr 27 05:33:14 s1 crmd: [6959]: info: process_te_message: Processing (N)ACK lrm_invoke-lrmd-1335533594-209 from s1
Apr 27 05:33:14 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_monitor_10000 (1) confirmed on s1 (rc=0)
Apr 27 05:33:14 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_monitor_10000 (call=46, status=1, cib-update=0, confirmed=true) Cancelled
Apr 27 05:33:14 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_stop_0 (9) confirmed on s2 (rc=0)
Apr 27 05:33:14 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 26 fired and confirmed
Apr 27 05:33:14 s1 crmd: [6959]: info: te_rsc_command: Initiating action 17: demote redis-serv-gr:1_demote_0 on s2
Apr 27 05:33:14 s1 crmd: [6959]: WARN: status_from_rc: Action 17 (redis-serv-gr:1_demote_0) on s2 failed (target: 0 vs. rc: 7): Error
Apr 27 05:33:14 s1 crmd: [6959]: WARN: update_failcount: Updating failcount for redis-serv-gr:1 on s2 after failed demote: rc=7 (update=value++, time=1335533594)
Apr 27 05:33:14 s1 crmd: [6959]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=redis-serv-gr:1_last_failure_0, magic=0:7;17:78:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77, cib=0.356.41) : Event failed
Apr 27 05:33:14 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Apr 27 05:33:14 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:33:14 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_demote_0 (17) confirmed on s2 (rc=4)
Apr 27 05:33:14 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Apr 27 05:33:14 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:33:14 s1 crmd: [6959]: notice: run_graph: Transition 78 (Complete=5, Pending=0, Fired=0, Skipped=11, Incomplete=3, Source=/var/lib/pengine/pe-input-1632.bz2): Stopped
Apr 27 05:33:14 s1 crmd: [6959]: info: te_graph_trigger: Transition 78 is now complete
Apr 27 05:33:14 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:33:14 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke: Query 215: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=215, ref=pe_calc-dc-1335533594-211, seq=364, quorate=1
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:14 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-s2-fail-count-redis-serv-gr.1, name=fail-count-redis-serv-gr:1, value=2, magic=NA, cib=0.356.42) : Transient attribute: update
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke: Query 216: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:14 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=216, ref=pe_calc-dc-1335533594-212, seq=364, quorate=1
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s2 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s2 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:0#011(Slave -> Master s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Stop    redis-serv-gr:1#011(s2)
Apr 27 05:33:14 s1 crmd: [6959]: info: handle_response: pe_calc calculation pe_calc-dc-1335533594-211 is obsolete
Apr 27 05:33:14 s1 pengine: [6958]: notice: process_pe_message: Transition 79: PEngine Input stored in: /var/lib/pengine/pe-input-1633.bz2
Apr 27 05:33:14 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:33:14 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Promote redis-serv-gr:0#011(Slave -> Master s1)
Apr 27 05:33:14 s1 pengine: [6958]: notice: LogActions: Stop    redis-serv-gr:1#011(s2)
Apr 27 05:33:14 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:33:14 s1 crmd: [6959]: WARN: destroy_action: Cancelling timer for action 1 (src=312)
Apr 27 05:33:14 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 80: 10 actions in 10 synapses
Apr 27 05:33:14 s1 crmd: [6959]: info: do_te_invoke: Processing graph 80 (ref=pe_calc-dc-1335533594-212) derived from /var/lib/pengine/pe-input-1634.bz2
Apr 27 05:33:14 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 16 fired and confirmed
Apr 27 05:33:14 s1 crmd: [6959]: info: te_rsc_command: Initiating action 2: stop redis-serv-gr:1_stop_0 on s2
Apr 27 05:33:14 s1 pengine: [6958]: notice: process_pe_message: Transition 80: PEngine Input stored in: /var/lib/pengine/pe-input-1634.bz2
Apr 27 05:33:16 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_stop_0 (2) confirmed on s2 (rc=0)
Apr 27 05:33:16 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 17 fired and confirmed
Apr 27 05:33:16 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 3 fired and confirmed
Apr 27 05:33:16 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 18 fired and confirmed
Apr 27 05:33:16 s1 crmd: [6959]: info: te_rsc_command: Initiating action 12: promote redis-serv-gr:0_promote_0 on s1 (local)
Apr 27 05:33:16 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=12:80:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_promote_0 )
Apr 27 05:33:16 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 promote[47] (pid 29070)
Apr 27 05:33:16 s1 crm_attribute: [29155]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:16 s1 attrd: [6957]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-redis-serv-gr:0 (10)
Apr 27 05:33:16 s1 attrd: [6957]: notice: attrd_perform_update: Sent update 129: master-redis-serv-gr:0=10
Apr 27 05:33:16 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-s1-master-redis-serv-gr.0, name=master-redis-serv-gr:0, value=10, magic=NA, cib=0.356.44) : Transient attribute: update
Apr 27 05:33:16 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Apr 27 05:33:16 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:33:16 s1 lrmd: [6956]: info: operation promote[47] on redis-serv-gr:0 for client 6959: pid 29070 exited with return code 0
Apr 27 05:33:16 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_promote_0 (call=47, rc=0, cib-update=217, confirmed=true) ok
Apr 27 05:33:16 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_promote_0 (12) confirmed on s1 (rc=0)
Apr 27 05:33:16 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 19 fired and confirmed
Apr 27 05:33:16 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:33:16 s1 crmd: [6959]: notice: run_graph: Transition 80 (Complete=7, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pengine/pe-input-1634.bz2): Stopped
Apr 27 05:33:16 s1 crmd: [6959]: info: te_graph_trigger: Transition 80 is now complete
Apr 27 05:33:16 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:33:16 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:33:16 s1 crmd: [6959]: info: do_pe_invoke: Query 218: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:16 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=218, ref=pe_calc-dc-1335533596-215, seq=364, quorate=1
Apr 27 05:33:16 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:16 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:16 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:16 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:33:16 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:33:16 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:33:16 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:16 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:16 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:33:16 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:33:16 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for ip-redis-gr on s1
Apr 27 05:33:16 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:16 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:16 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (5s) for redis-serv-gr:0 on s1
Apr 27 05:33:16 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:16 s1 pengine: [6958]: notice: LogActions: Start   ip-redis-gr#011(s1)
Apr 27 05:33:16 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:0#011(Master s1)
Apr 27 05:33:16 s1 pengine: [6958]: notice: LogActions: Start   redis-serv-gr:1#011(s2)
Apr 27 05:33:16 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:33:16 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 81: 7 actions in 7 synapses
Apr 27 05:33:16 s1 crmd: [6959]: info: do_te_invoke: Processing graph 81 (ref=pe_calc-dc-1335533596-215) derived from /var/lib/pengine/pe-input-1635.bz2
Apr 27 05:33:16 s1 crmd: [6959]: info: te_rsc_command: Initiating action 5: start ip-redis-gr_start_0 on s1 (local)
Apr 27 05:33:16 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=5:81:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=ip-redis-gr_start_0 )
Apr 27 05:33:16 s1 lrmd: [6956]: info: rsc:ip-redis-gr start[48] (pid 29156)
Apr 27 05:33:16 s1 crmd: [6959]: info: te_rsc_command: Initiating action 11: monitor redis-serv-gr:0_monitor_5000 on s1 (local)
Apr 27 05:33:16 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=11:81:8:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=redis-serv-gr:0_monitor_5000 )
Apr 27 05:33:16 s1 lrmd: [6956]: info: rsc:redis-serv-gr:0 monitor[49] (pid 29157)
Apr 27 05:33:16 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 14 fired and confirmed
Apr 27 05:33:16 s1 crmd: [6959]: info: te_rsc_command: Initiating action 12: start redis-serv-gr:1_start_0 on s2
Apr 27 05:33:16 s1 pengine: [6958]: notice: process_pe_message: Transition 81: PEngine Input stored in: /var/lib/pengine/pe-input-1635.bz2
Apr 27 05:33:17 s1 IPaddr2[29156]: INFO: ip -f inet addr add 192.168.1.15/24 brd 192.168.1.255 dev eth0
Apr 27 05:33:17 s1 IPaddr2[29156]: INFO: ip link set eth0 up
Apr 27 05:33:17 s1 IPaddr2[29156]: INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.1.15 eth0 192.168.1.15 auto not_used not_used
Apr 27 05:33:17 s1 lrmd: [6956]: info: operation start[48] on ip-redis-gr for client 6959: pid 29156 exited with return code 0
Apr 27 05:33:17 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_start_0 (call=48, rc=0, cib-update=219, confirmed=true) ok
Apr 27 05:33:17 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_start_0 (5) confirmed on s1 (rc=0)
Apr 27 05:33:17 s1 crmd: [6959]: info: te_rsc_command: Initiating action 6: monitor ip-redis-gr_monitor_10000 on s1 (local)
Apr 27 05:33:17 s1 crmd: [6959]: info: do_lrm_rsc_op: Performing key=6:81:0:b63cb1ed-d42c-456e-b4b0-3c5da4b81a77 op=ip-redis-gr_monitor_10000 )
Apr 27 05:33:17 s1 lrmd: [6956]: info: rsc:ip-redis-gr monitor[50] (pid 29243)
Apr 27 05:33:17 s1 lrmd: [6956]: info: operation monitor[50] on ip-redis-gr for client 6959: pid 29243 exited with return code 0
Apr 27 05:33:17 s1 crmd: [6959]: info: process_lrm_event: LRM operation ip-redis-gr_monitor_10000 (call=50, rc=0, cib-update=220, confirmed=false) ok
Apr 27 05:33:17 s1 crmd: [6959]: info: match_graph_event: Action ip-redis-gr_monitor_10000 (6) confirmed on s1 (rc=0)
Apr 27 05:33:17 s1 crm_attribute: [29287]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:17 s1 lrmd: [6956]: info: operation monitor[49] on redis-serv-gr:0 for client 6959: pid 29157 exited with return code 8
Apr 27 05:33:17 s1 crmd: [6959]: info: process_lrm_event: LRM operation redis-serv-gr:0_monitor_5000 (call=49, rc=8, cib-update=221, confirmed=false) master
Apr 27 05:33:17 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:0_monitor_5000 (11) confirmed on s1 (rc=0)
Apr 27 05:33:17 s1 crmd: [6959]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=0, tag=nvpair, id=status-s2-master-redis-serv-gr.1, name=master-redis-serv-gr:1, value=5, magic=NA, cib=0.356.49) : Transient attribute: update
Apr 27 05:33:17 s1 crmd: [6959]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
Apr 27 05:33:17 s1 crmd: [6959]: info: update_abort_priority: Abort action done superceeded by restart
Apr 27 05:33:17 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_start_0 (12) confirmed on s2 (rc=0)
Apr 27 05:33:17 s1 crmd: [6959]: info: te_pseudo_action: Pseudo action 15 fired and confirmed
Apr 27 05:33:17 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:33:17 s1 crmd: [6959]: notice: run_graph: Transition 81 (Complete=6, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-1635.bz2): Stopped
Apr 27 05:33:17 s1 crmd: [6959]: info: te_graph_trigger: Transition 81 is now complete
Apr 27 05:33:17 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:33:17 s1 crmd: [6959]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.
Apr 27 05:33:17 s1 crmd: [6959]: info: do_pe_invoke: Query 222: Requesting the current CIB: S_POLICY_ENGINE
Apr 27 05:33:17 s1 crmd: [6959]: info: do_pe_invoke_callback: Invoking the PE: query=222, ref=pe_calc-dc-1335533597-220, seq=364, quorate=1
Apr 27 05:33:17 s1 pengine: [6958]: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 27 05:33:17 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:0_last_failure_0 on s1: not running (7)
Apr 27 05:33:17 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:0 to stop after a failed demote action
Apr 27 05:33:17 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_monitor_5000 on s2: master (failed) (9)
Apr 27 05:33:17 s1 pengine: [6958]: WARN: unpack_rsc_op: Processing failed op redis-serv-gr:1_last_failure_0 on s2: not running (7)
Apr 27 05:33:17 s1 pengine: [6958]: WARN: unpack_rsc_op: Forcing redis-serv-gr:1 to stop after a failed demote action
Apr 27 05:33:17 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:17 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999999 more times on s1 before being forced off
Apr 27 05:33:17 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:33:17 s1 pengine: [6958]: notice: common_apply_stickiness: redis-gr can fail 999998 more times on s2 before being forced off
Apr 27 05:33:17 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:17 s1 pengine: [6958]: notice: RecurringOp:  Start recurring monitor (10s) for redis-serv-gr:1 on s2
Apr 27 05:33:17 s1 pengine: [6958]: notice: LogActions: Leave   ip-redis-gr#011(Started s1)
Apr 27 05:33:17 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:0#011(Master s1)
Apr 27 05:33:17 s1 pengine: [6958]: notice: LogActions: Leave   redis-serv-gr:1#011(Slave s2)
Apr 27 05:33:17 s1 crmd: [6959]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Apr 27 05:33:17 s1 crmd: [6959]: info: unpack_graph: Unpacked transition 82: 1 actions in 1 synapses
Apr 27 05:33:17 s1 crmd: [6959]: info: do_te_invoke: Processing graph 82 (ref=pe_calc-dc-1335533597-220) derived from /var/lib/pengine/pe-input-1636.bz2
Apr 27 05:33:17 s1 crmd: [6959]: info: te_rsc_command: Initiating action 16: monitor redis-serv-gr:1_monitor_10000 on s2
Apr 27 05:33:18 s1 pengine: [6958]: notice: process_pe_message: Transition 82: PEngine Input stored in: /var/lib/pengine/pe-input-1636.bz2
Apr 27 05:33:18 s1 crmd: [6959]: info: match_graph_event: Action redis-serv-gr:1_monitor_10000 (16) confirmed on s2 (rc=0)
Apr 27 05:33:18 s1 crmd: [6959]: info: run_graph: ====================================================
Apr 27 05:33:18 s1 crmd: [6959]: notice: run_graph: Transition 82 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-1636.bz2): Complete
Apr 27 05:33:18 s1 crmd: [6959]: info: te_graph_trigger: Transition 82 is now complete
Apr 27 05:33:18 s1 crmd: [6959]: info: notify_crmd: Transition 82 status: done - <null>
Apr 27 05:33:18 s1 crmd: [6959]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Apr 27 05:33:18 s1 crmd: [6959]: info: do_state_transition: Starting PEngine Recheck Timer
Apr 27 05:33:22 s1 crm_attribute: [29341]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:28 s1 crm_attribute: [29422]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:33 s1 crm_attribute: [29475]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:38 s1 crm_attribute: [29556]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:44 s1 crm_attribute: [29609]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:49 s1 crm_attribute: [29690]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:54 s1 crm_attribute: [29811]: info: Invoked: crm_attribute -N s1 -n master-redis-serv-gr:0 -l reboot -v 10 
Apr 27 05:33:55 s1 logger: Mark:HB_REPORT:1335533633
