[Pacemaker] Stuck in a STONITH cycle

Tue Oct 16 00:03:58 EDT 2012

On Sat, Oct 13, 2012 at 7:58 AM, David Parker <dparker at utica.edu> wrote:
> Hello,
>
> I have two nodes set up in a cluster to provide a MySQL server (mysqld) in
> HA on a virtual IP address.  This was working fine until I had to reboot the
> servers.  All I did was change the interface each node uses for its primary
> IP address (changed from eth1 to eth0 on each node).  Now I'm stuck in a
> cycle.  Let's say node 1 has the virtual IP and is running mysqld, and node
> 2 is down.  When node 2 boots up, it will STONITH node 1 for no apparent
> reason

At a guess... because it still cant reach node 1?

> and take over the resources, which shouldn't happen.

Why?  This would be expected in a 2-node cluster where either side
comes up while the network is hosed.

What you have is a stonith death match: http://ourobengr.com/ha/

>  When node 1
> boots up again, it will STONITH node 2 and take over the resources, which
> again shouldn't happen.
>
> Obviously, the goal is to have both nodes running, and whichever one boots
> up second should see that the first one has the resources and then stand by.
> I'll include the relevant log entries and my crm config at the end of this
> message.  In the logs, the only thing I see that seems unusual (other than
> the fact that a STONITH is executed against a running node) is this:
>
> Oct 12 16:27:22 ha1 crmd: [1176]: info: populate_cib_nodes_ha: Requesting
> the list of configured nodes
> Oct 12 16:27:23 ha1 crmd: [1176]: WARN: get_uuid: Could not calculate UUID
> for ha2
> Oct 12 16:27:23 ha1 crmd: [1176]: WARN: populate_cib_nodes_ha: Node ha2: no
> uuid found
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_state_transition: All 1 cluster
> nodes are eligible to run resources.
>
> The exact opposite shows up on the node "ha2" (it says ha1 has no uuid).
> Did the UUID of each node change because the physical interface changed?
> Any other ideas?  Thanks in advance.
>
>     - Dave
>
> Here are the log lines:
>
> Oct 12 16:26:13 ha1 heartbeat: [1176]: info: Starting
> "/usr/lib/heartbeat/crmd" as uid 103  gid 105 (pid 1176)
> Oct 12 16:26:13 ha1 crmd: [1176]: info: crm_log_init: Changed active
> directory to /var/lib/heartbeat/cores/hacluster
> Oct 12 16:26:13 ha1 crmd: [1176]: info: Invoked: /usr/lib/heartbeat/crmd
> Oct 12 16:26:13 ha1 crmd: [1176]: info: main: CRM Hg Version:
> 74392a28b7f31d7ddc86689598bd23114f58978b
> Oct 12 16:26:13 ha1 crmd: [1176]: info: crmd_init: Starting crmd
> Oct 12 16:26:13 ha1 crmd: [1176]: info: G_main_add_SignalHandler: Added
> signal handler for signal 17
> Oct 12 16:26:14 ha1 crmd: [1176]: info: do_cib_control: CIB connection
> established
> Oct 12 16:26:14 ha1 crmd: [1176]: info: crm_cluster_connect: Connecting to
> Heartbeat
> Oct 12 16:26:15 ha1 crmd: [1176]: info: register_heartbeat_conn: Hostname:
> ha1
> Oct 12 16:26:15 ha1 crmd: [1176]: info: register_heartbeat_conn: UUID:
> 1b48f410-44d1-4e89-8b52-ff23b32db1bc
> Oct 12 16:26:15 ha1 crmd: [1176]: info: do_ha_control: Connected to the
> cluster
> Oct 12 16:26:15 ha1 crmd: [1176]: info: do_ccm_control: CCM connection
> established... waiting for first callback
> Oct 12 16:26:15 ha1 crmd: [1176]: info: do_started: Delaying start, CCM
> (0000000000100000) not connected
> Oct 12 16:26:15 ha1 crmd: [1176]: info: crmd_init: Starting crmd's mainloop
> Oct 12 16:26:15 ha1 crmd: [1176]: info: config_query_callback: Checking for
> expired actions every 900000ms
> Oct 12 16:26:15 ha1 crmd: [1176]: notice: crmd_client_status_callback:
> Status update: Client ha1/crmd now has status [online] (DC=false)
> Oct 12 16:26:15 ha1 crmd: [1176]: info: crm_new_peer: Node 0 is now known as
> ha1
> Oct 12 16:26:15 ha1 crmd: [1176]: info: crm_update_peer_proc: ha1.crmd is
> now online
> Oct 12 16:26:15 ha1 crmd: [1176]: info: crmd_client_status_callback: Not the
> DC
> Oct 12 16:26:15 ha1 crmd: [1176]: notice: crmd_client_status_callback:
> Status update: Client ha1/crmd now has status [online] (DC=false)
> Oct 12 16:26:16 ha1 crmd: [1176]: info: crmd_client_status_callback: Not the
> DC
> Oct 12 16:26:16 ha1 crmd: [1176]: info: mem_handle_event: Got an event
> OC_EV_MS_NEW_MEMBERSHIP from ccm
> Oct 12 16:26:16 ha1 crmd: [1176]: info: mem_handle_event: instance=1,
> nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
> Oct 12 16:26:16 ha1 crmd: [1176]: info: crmd_ccm_msg_callback: Quorum
> (re)attained after event=NEW MEMBERSHIP (id=1)
> Oct 12 16:26:16 ha1 crmd: [1176]: info: ccm_event_detail: NEW MEMBERSHIP:
> trans=1, nodes=1, new=1, lost=0 n_idx=0, new_idx=0, old_idx=3
> Oct 12 16:26:16 ha1 crmd: [1176]: info: ccm_event_detail: #011CURRENT: ha1
> [nodeid=0, born=1]
> Oct 12 16:26:16 ha1 crmd: [1176]: info: ccm_event_detail: #011NEW:     ha1
> [nodeid=0, born=1]
> Oct 12 16:26:16 ha1 crmd: [1176]: info: crm_update_peer: Node ha1: id=0
> state=member (new) addr=(null) votes=-1 born=1 seen=1
> proc=00000000000000000000000000000200
> Oct 12 16:26:16 ha1 crmd: [1176]: info: crm_update_peer_proc: ha1.ais is now
> online
> Oct 12 16:26:16 ha1 crmd: [1176]: info: do_started: The local CRM is
> operational
> Oct 12 16:26:16 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> origin=do_started ]
> Oct 12 16:27:17 ha1 crmd: [1176]: info: crm_timer_popped: Election Trigger
> (I_DC_TIMEOUT) just popped!
> Oct 12 16:27:17 ha1 crmd: [1176]: WARN: do_log: FSA: Input I_DC_TIMEOUT from
> crm_timer_popped() received in state S_PENDING
> Oct 12 16:27:17 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED
> origin=crm_timer_popped ]
> Oct 12 16:27:17 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Oct 12 16:27:17 ha1 crmd: [1176]: info: do_te_control: Registering TE UUID:
> 4f352feb-6be6-4af1-bb9e-af8b7ca28f58
> Oct 12 16:27:17 ha1 crmd: [1176]: WARN: cib_client_add_notify_callback:
> Callback already present
> Oct 12 16:27:17 ha1 crmd: [1176]: info: set_graph_functions: Setting custom
> graph functions
> Oct 12 16:27:17 ha1 crmd: [1176]: info: unpack_graph: Unpacked transition
> -1: 0 actions in 0 synapses
> Oct 12 16:27:17 ha1 crmd: [1176]: info: start_subsystem: Starting sub-system
> "pengine"
> Oct 12 16:27:20 ha1 crmd: [1176]: info: do_dc_takeover: Taking over DC
> status for this partition
> Oct 12 16:27:20 ha1 crmd: [1176]: info: join_make_offer: Making join offers
> based on membership 1
> Oct 12 16:27:20 ha1 crmd: [1176]: info: do_dc_join_offer_all: join-1:
> Waiting on 1 outstanding join acks
> Oct 12 16:27:20 ha1 crmd: [1176]: info: te_connect_stonith: Attempting
> connection to fencing daemon...
> Oct 12 16:27:21 ha1 crmd: [1176]: info: te_connect_stonith: Connected
> Oct 12 16:27:21 ha1 crmd: [1176]: info: config_query_callback: Checking for
> expired actions every 900000ms
> Oct 12 16:27:21 ha1 crmd: [1176]: info: update_dc: Set DC to ha1 (3.0.1)
> Oct 12 16:27:21 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
> cause=C_FSA_INTERNAL origin=check_join_state ]
> Oct 12 16:27:21 ha1 crmd: [1176]: info: do_state_transition: All 1 cluster
> nodes responded to the join offer.
> Oct 12 16:27:21 ha1 crmd: [1176]: info: do_dc_join_finalize: join-1: Syncing
> the CIB from ha1 to the rest of the cluster
> Oct 12 16:27:22 ha1 crmd: [1176]: info: update_attrd: Connecting to attrd...
> Oct 12 16:27:22 ha1 crmd: [1176]: info: erase_xpath_callback: Deletion of
> "//node_state[@uname='ha1']/transient_attributes": ok (rc=0)
> Oct 12 16:27:22 ha1 crmd: [1176]: info: do_dc_join_ack: join-1: Updating
> node state to member for ha1
> Oct 12 16:27:22 ha1 crmd: [1176]: info: erase_xpath_callback: Deletion of
> "//node_state[@uname='ha1']/lrm": ok (rc=0)
> Oct 12 16:27:22 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
> cause=C_FSA_INTERNAL origin=check_join_state ]
> Oct 12 16:27:22 ha1 crmd: [1176]: info: populate_cib_nodes_ha: Requesting
> the list of configured nodes
> Oct 12 16:27:23 ha1 crmd: [1176]: WARN: get_uuid: Could not calculate UUID
> for ha2
> Oct 12 16:27:23 ha1 crmd: [1176]: WARN: populate_cib_nodes_ha: Node ha2: no
> uuid found
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_state_transition: All 1 cluster
> nodes are eligible to run resources.
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_dc_join_final: Ensuring DC,
> quorum and node attributes are up-to-date
> Oct 12 16:27:23 ha1 crmd: [1176]: info: crm_update_quorum: Updating quorum
> status to true (call=21)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: abort_transition_graph:
> do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_pe_invoke: Query 22: Requesting
> the current CIB: S_POLICY_ENGINE
> Oct 12 16:27:23 ha1 crmd: [1176]: info: abort_transition_graph:
> need_abort:59 - Triggered transition abort (complete=1) : Non-status change
> Oct 12 16:27:23 ha1 crmd: [1176]: info: need_abort: Aborting on change to
> admin_epoch
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_pe_invoke: Query 23: Requesting
> the current CIB: S_POLICY_ENGINE
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_pe_invoke_callback: Invoking the
> PE: query=23, ref=pe_calc-dc-1350073643-7, seq=1, quorate=1
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=handle_response ]
> Oct 12 16:27:23 ha1 crmd: [1176]: info: unpack_graph: Unpacked transition 0:
> 20 actions in 20 synapses
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_te_invoke: Processing graph 0
> (ref=pe_calc-dc-1350073643-7) derived from /var/lib/pengine/pe-warn-1370.bz2
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 4:
> monitor stonith-ha1_monitor_0 on ha1 (local)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=4:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=stonith-ha1_monitor_0 )
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 5:
> monitor stonith-ha2_monitor_0 on ha1 (local)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=5:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=stonith-ha2_monitor_0 )
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 6:
> monitor MysqlIP_monitor_0 on ha1 (local)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=6:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=MysqlIP_monitor_0 )
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 7:
> monitor mysqld_monitor_0 on ha1 (local)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=7:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=mysqld_monitor_0 )
> Oct 12 16:27:23 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> stonith-ha1_monitor_0 (call=2, rc=7, cib-update=24, confirmed=true) not
> running
> Oct 12 16:27:23 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> stonith-ha2_monitor_0 (call=3, rc=7, cib-update=25, confirmed=true) not
> running
> Oct 12 16:27:23 ha1 crmd: [1176]: info: match_graph_event: Action
> stonith-ha1_monitor_0 (4) confirmed on ha1 (rc=0)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: match_graph_event: Action
> stonith-ha2_monitor_0 (5) confirmed on ha1 (rc=0)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> mysqld_monitor_0 (call=5, rc=7, cib-update=26, confirmed=true) not running
> Oct 12 16:27:23 ha1 crmd: [1176]: info: match_graph_event: Action
> mysqld_monitor_0 (7) confirmed on ha1 (rc=0)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> MysqlIP_monitor_0 (call=4, rc=7, cib-update=27, confirmed=true) not running
> Oct 12 16:27:23 ha1 crmd: [1176]: info: match_graph_event: Action
> MysqlIP_monitor_0 (6) confirmed on ha1 (rc=0)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 3:
> probe_complete probe_complete on ha1 (local) - no waiting
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 2
> fired and confirmed
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 8:
> start stonith-ha2_start_0 on ha1 (local)
> Oct 12 16:27:23 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=8:0:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=stonith-ha2_start_0 )
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 14
> fired and confirmed
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 20
> fired and confirmed
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 19
> fired and confirmed
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 15
> fired and confirmed
> Oct 12 16:27:23 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 12
> fired and confirmed
> Oct 12 16:27:23 ha1 crmd: [1176]: info: abort_transition_graph:
> te_update_diff:146 - Triggered transition abort (complete=0,
> tag=transient_attributes, id=1b48f410-44d1-4e89-8b52-ff23b32db1bc, magic=NA,
> cib=0.61.6) : Transient attribute:
>  update
> Oct 12 16:27:23 ha1 crmd: [1176]: info: update_abort_priority: Abort
> priority upgraded from 0 to 1000000
> Oct 12 16:27:23 ha1 crmd: [1176]: info: update_abort_priority: Abort action
> done superceeded by restart
> Oct 12 16:27:34 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> stonith-ha2_start_0 (call=6, rc=0, cib-update=28, confirmed=true) ok
> Oct 12 16:27:34 ha1 crmd: [1176]: info: match_graph_event: Action
> stonith-ha2_start_0 (8) confirmed on ha1 (rc=0)
> Oct 12 16:27:34 ha1 crmd: [1176]: info: run_graph:
> ====================================================
> Oct 12 16:27:34 ha1 crmd: [1176]: notice: run_graph: Transition 0
> (Complete=12, Pending=0, Fired=0, Skipped=8, Incomplete=0,
> Source=/var/lib/pengine/pe-warn-1370.bz2): Stopped
> Oct 12 16:27:34 ha1 crmd: [1176]: info: te_graph_trigger: Transition 0 is
> now complete
> Oct 12 16:27:34 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Oct 12 16:27:34 ha1 crmd: [1176]: info: do_state_transition: All 1 cluster
> nodes are eligible to run resources.
> Oct 12 16:27:34 ha1 crmd: [1176]: info: do_pe_invoke: Query 29: Requesting
> the current CIB: S_POLICY_ENGINE
> Oct 12 16:27:34 ha1 crmd: [1176]: info: do_pe_invoke_callback: Invoking the
> PE: query=29, ref=pe_calc-dc-1350073654-14, seq=1, quorate=1
> Oct 12 16:27:34 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=handle_response ]
> Oct 12 16:27:34 ha1 crmd: [1176]: info: unpack_graph: Unpacked transition 1:
> 9 actions in 9 synapses
> Oct 12 16:27:34 ha1 crmd: [1176]: info: do_te_invoke: Processing graph 1
> (ref=pe_calc-dc-1350073654-14) derived from
> /var/lib/pengine/pe-warn-1371.bz2
> Oct 12 16:27:34 ha1 stonithd: [1174]: info: client tengine [pid: 1176]
> requests a STONITH operation RESET on node ha2
> Oct 12 16:27:34 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 9
> fired and confirmed
> Oct 12 16:27:34 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 13
> fired and confirmed
> Oct 12 16:27:34 ha1 crmd: [1176]: info: te_fence_node: Executing reboot
> fencing operation (15) on ha2 (timeout=60000)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: tengine_stonith_callback: call=1266,
> optype=1, node_name=ha2, result=0, node_list=ha1,
> action=15:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 1
> fired and confirmed
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 14
> fired and confirmed
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 6:
> start MysqlIP_start_0 on ha1 (local)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=6:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=MysqlIP_start_0 )
> Oct 12 16:28:08 ha1 crmd: [1176]: info: erase_xpath_callback: Deletion of
> "//node_state[@uname='ha2']/lrm": ok (rc=0)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: erase_xpath_callback: Deletion of
> "//node_state[@uname='ha2']/transient_attributes": ok (rc=0)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> MysqlIP_start_0 (call=7, rc=0, cib-update=33, confirmed=true) ok
> Oct 12 16:28:08 ha1 crmd: [1176]: info: match_graph_event: Action
> MysqlIP_start_0 (6) confirmed on ha1 (rc=0)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 7:
> monitor MysqlIP_monitor_10000 on ha1 (local)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=7:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=MysqlIP_monitor_10000 )
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_rsc_command: Initiating action 8:
> start mysqld_start_0 on ha1 (local)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: do_lrm_rsc_op: Performing
> key=8:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58 op=mysqld_start_0 )
> Oct 12 16:28:08 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> mysqld_start_0 (call=9, rc=0, cib-update=34, confirmed=true) ok
> Oct 12 16:28:08 ha1 crmd: [1176]: info: match_graph_event: Action
> mysqld_start_0 (8) confirmed on ha1 (rc=0)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_pseudo_action: Pseudo action 10
> fired and confirmed
> Oct 12 16:28:08 ha1 crmd: [1176]: info: process_lrm_event: LRM operation
> MysqlIP_monitor_10000 (call=8, rc=0, cib-update=35, confirmed=false) ok
> Oct 12 16:28:08 ha1 crmd: [1176]: info: match_graph_event: Action
> MysqlIP_monitor_10000 (7) confirmed on ha1 (rc=0)
> Oct 12 16:28:08 ha1 crmd: [1176]: info: run_graph:
> ====================================================
> Oct 12 16:28:08 ha1 crmd: [1176]: notice: run_graph: Transition 1
> (Complete=9, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pengine/pe-warn-1371.bz2): Complete
> Oct 12 16:28:08 ha1 crmd: [1176]: info: te_graph_trigger: Transition 1 is
> now complete
> Oct 12 16:28:08 ha1 crmd: [1176]: info: notify_crmd: Transition 1 status:
> done - <null>
> Oct 12 16:28:08 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Oct 12 16:28:08 ha1 crmd: [1176]: info: do_state_transition: Starting
> PEngine Recheck Timer
> Oct 12 16:43:08 ha1 crmd: [1176]: info: crm_timer_popped: PEngine Recheck
> Timer (I_PE_CALC) just popped!
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
> origin=crm_timer_popped ]
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_state_transition: Progressed to
> state S_POLICY_ENGINE after C_TIMER_POPPED
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_state_transition: All 1 cluster
> nodes are eligible to run resources.
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_pe_invoke: Query 36: Requesting
> the current CIB: S_POLICY_ENGINE
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_pe_invoke_callback: Invoking the
> PE: query=36, ref=pe_calc-dc-1350074588-18, seq=1, quorate=1
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
> cause=C_IPC_MESSAGE origin=handle_response ]
> Oct 12 16:43:08 ha1 crmd: [1176]: info: unpack_graph: Unpacked transition 2:
> 0 actions in 0 synapses
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_te_invoke: Processing graph 2
> (ref=pe_calc-dc-1350074588-18) derived from
> /var/lib/pengine/pe-input-4140.bz2
> Oct 12 16:43:08 ha1 crmd: [1176]: info: run_graph:
> ====================================================
> Oct 12 16:43:08 ha1 crmd: [1176]: notice: run_graph: Transition 2
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pengine/pe-input-4140.bz2): Complete
> Oct 12 16:43:08 ha1 crmd: [1176]: info: te_graph_trigger: Transition 2 is
> now complete
> Oct 12 16:43:08 ha1 crmd: [1176]: info: notify_crmd: Transition 2 status:
> done - <null>
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Oct 12 16:43:08 ha1 crmd: [1176]: info: do_state_transition: Starting
> PEngine Recheck Timer
>
> Here's my config:
>
> root at ha1:~# cibadmin --query
> <cib validate-with="pacemaker-1.0" crm_feature_set="3.0.1" have-quorum="1"
> admin_epoch="0" epoch="61" num_updates="11" cib-last-written="Fri Oct 12
> 16:07:33 2012" dc-uuid="1b48f410-44d1-4e89-8b52-ff23b32db1bc">
> <configuration>
> <crm_config>
> <cluster_property_set id="cib-bootstrap-options">
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
> value="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b"/>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure"
> name="cluster-infrastructure" value="Heartbeat"/>
> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled"
> value="true"/>
> </cluster_property_set>
> </crm_config>
> <nodes>
> <node id="1b48f410-44d1-4e89-8b52-ff23b32db1bc" uname="ha1" type="normal"/>
> <node id="9790fe6e-67b2-4817-abf4-966b5aa6948c" uname="ha2" type="normal"/>
> </nodes>
> <resources>
> <primitive class="stonith" id="stonith-ha1" type="external/riloe">
> <instance_attributes id="stonith-ha1-instance_attributes">
> <nvpair id="stonith-ha1-instance_attributes-hostlist" name="hostlist"
> value="ha1"/>
> <nvpair id="stonith-ha1-instance_attributes-ilo_hostname"
> name="ilo_hostname" value="10.0.1.111"/>
> <nvpair id="stonith-ha1-instance_attributes-ilo_user" name="ilo_user"
> value="Administrator"/>
> <nvpair id="stonith-ha1-instance_attributes-ilo_password"
> name="ilo_password" value="xxx"/>
> <nvpair id="stonith-ha1-instance_attributes-ilo_can_reset"
> name="ilo_can_reset" value="1"/>
> <nvpair id="stonith-ha1-instance_attributes-ilo_protocol"
> name="ilo_protocol" value="2.0"/>
> <nvpair id="stonith-ha1-instance_attributes-ilo_powerdown_method"
> name="ilo_powerdown_method" value="button"/>
> </instance_attributes>
> </primitive>
> <primitive class="stonith" id="stonith-ha2" type="external/riloe">
> <instance_attributes id="stonith-ha2-instance_attributes">
> <nvpair id="stonith-ha2-instance_attributes-hostlist" name="hostlist"
> value="ha2"/>
> <nvpair id="stonith-ha2-instance_attributes-ilo_hostname"
> name="ilo_hostname" value="10.0.1.112"/>
> <nvpair id="stonith-ha2-instance_attributes-ilo_user" name="ilo_user"
> value="Administrator"/>
> <nvpair id="stonith-ha2-instance_attributes-ilo_password"
> name="ilo_password" value="xxx"/>
> <nvpair id="stonith-ha2-instance_attributes-ilo_can_reset"
> name="ilo_can_reset" value="1"/>
> <nvpair id="stonith-ha2-instance_attributes-ilo_protocol"
> name="ilo_protocol" value="2.0"/>
> <nvpair id="stonith-ha2-instance_attributes-ilo_powerdown_method"
> name="ilo_powerdown_method" value="button"/>
> </instance_attributes>
> </primitive>
> <group id="mysql-resources">
> <primitive class="ocf" id="MysqlIP" provider="heartbeat" type="IPaddr2">
> <instance_attributes id="MysqlIP-instance_attributes">
> <nvpair id="MysqlIP-instance_attributes-ip" name="ip" value="192.168.25.9"/>
> <nvpair id="MysqlIP-instance_attributes-cidr_netmask" name="cidr_netmask"
> value="32"/>
> </instance_attributes>
> <operations>
> <op id="MysqlIP-monitor-10s" interval="10s" name="monitor"/>
> </operations>
> </primitive>
> <primitive id="mysqld" class="lsb" type="mysqld"/>
> </group>
> </resources>
> <constraints>
> <rsc_location id="loc-3" rsc="stonith-ha1" node="ha1" score="-INFINITY"/>
> <rsc_location id="loc-4" rsc="stonith-ha2" node="ha2" score="-INFINITY"/>
> <rsc_location id="loc-5" rsc="mysql-resources" node="ha1" score="100"/>
> </constraints>
> <rsc_defaults/>
> <op_defaults/>
> </configuration>
> <status>
> <node_state id="1b48f410-44d1-4e89-8b52-ff23b32db1bc" uname="ha1"
> ha="active" in_ccm="true" crmd="online" join="member" expected="member"
> crm-debug-origin="do_update_resource" shutdown="0">
> <lrm id="1b48f410-44d1-4e89-8b52-ff23b32db1bc">
> <lrm_resources>
> <lrm_resource id="stonith-ha1" type="external/riloe" class="stonith">
> <lrm_rsc_op id="stonith-ha1_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="4:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:7;4:0:7:4f352feb-6be6-4
> af1-bb9e-af8b7ca28f58" call-id="2" rc-code="7" op-status="0" interval="0"
> last-run="1350073642" last-rc-change="1350073642" exec-time="10"
> queue-time="0" op-digest="f7c04bf4ee3eb60cf4f9e841449a0fe6"/>
> </lrm_resource>
> <lrm_resource id="stonith-ha2" type="external/riloe" class="stonith">
> <lrm_rsc_op id="stonith-ha2_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="5:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:7;5:0:7:4f352feb-6be6-4
> af1-bb9e-af8b7ca28f58" call-id="3" rc-code="7" op-status="0" interval="0"
> last-run="1350073643" last-rc-change="1350073643" exec-time="0"
> queue-time="0" op-digest="c8773e5caa7ae2a75f71a79adc582d54"/>
> <lrm_rsc_op id="stonith-ha2_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="8:0:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:0;8:0:0:4f352feb-6be6-4af1-
> bb9e-af8b7ca28f58" call-id="6" rc-code="0" op-status="0" interval="0"
> last-run="1350073643" last-rc-change="1350073643" exec-time="10920"
> queue-time="0" op-digest="c8773e5caa7ae2a75f71a79adc582d54"/>
> </lrm_resource>
> <lrm_resource id="mysqld" type="mysqld" class="lsb">
> <lrm_rsc_op id="mysqld_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="7:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:7;7:0:7:4f352feb-6be6-4af1-b
> b9e-af8b7ca28f58" call-id="5" rc-code="7" op-status="0" interval="0"
> last-run="1350073642" last-rc-change="1350073642" exec-time="40"
> queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
> <lrm_rsc_op id="mysqld_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="8:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:0;8:1:0:4f352feb-6be6-4af1-bb9e-
> af8b7ca28f58" call-id="9" rc-code="0" op-status="0" interval="0"
> last-run="1350073688" last-rc-change="1350073688" exec-time="0"
> queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
> </lrm_resource>
> <lrm_resource id="MysqlIP" type="IPaddr2" class="ocf" provider="heartbeat">
> <lrm_rsc_op id="MysqlIP_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="6:0:7:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:7;6:0:7:4f352feb-6be6-4af1-
> bb9e-af8b7ca28f58" call-id="4" rc-code="7" op-status="0" interval="0"
> last-run="1350073642" last-rc-change="1350073642" exec-time="150"
> queue-time="0" op-digest="9611b7026c2dc135fbd13d3537b42d16"/>
> <lrm_rsc_op id="MysqlIP_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="6:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:0;6:1:0:4f352feb-6be6-4af1-bb9e
> -af8b7ca28f58" call-id="7" rc-code="0" op-status="0" interval="0"
> last-run="1350073687" last-rc-change="1350073687" exec-time="30"
> queue-time="0" op-digest="9611b7026c2dc135fbd13d3537b42d16"/>
> <lrm_rsc_op id="MysqlIP_monitor_10000" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
> transition-key="7:1:0:4f352feb-6be6-4af1-bb9e-af8b7ca28f58"
> transition-magic="0:0;7:1:0:4f352feb-6be6-4
> af1-bb9e-af8b7ca28f58" call-id="8" rc-code="0" op-status="0"
> interval="10000" last-run="1350073687" last-rc-change="1350073687"
> exec-time="10" queue-time="0" op-digest="caf3454e2233540200e7a6a5319c5362"/>
> </lrm_resource>
> </lrm_resources>
> </lrm>
> <transient_attributes id="1b48f410-44d1-4e89-8b52-ff23b32db1bc">
> <instance_attributes id="status-1b48f410-44d1-4e89-8b52-ff23b32db1bc">
> <nvpair id="status-1b48f410-44d1-4e89-8b52-ff23b32db1bc-probe_complete"
> name="probe_complete" value="true"/>
> </instance_attributes>
> </transient_attributes>
> </node_state>
> <node_state id="9790fe6e-67b2-4817-abf4-966b5aa6948c" uname="ha2" ha="dead"
> in_ccm="false" crmd="offline" join="down" expected="down"
> crm-debug-origin="send_stonith_update"/>
> </status>
> </cib>
>
> --
>
> Dave Parker
> Systems Administrator
> Utica College
> Integrated Information Technology Services
> (315) 792-3229
> Registered Linux User #408177
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org