[Pacemaker] stonith device is failing to start
Shravan Mishra
shravan.mishra at gmail.com
Wed Dec 2 23:47:59 UTC 2009
Hi guys,
We are testing Active passive scenario. We pulled out the power cord from
primary and expected to see secondary takeover as primary but this is what
we see:
On doing crm_mon I see :
============
============
Last updated: Wed Dec 2 17:36:40 2009
Stack: openais
Current DC: IP133.itactics.com - partition WITHOUT quorum
Version: 1.0.5-9e9faaab40f3f97e3c0d623e4a4c47ed83fa1601
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Node IP131.itactics.com: UNCLEAN (offline)
Online: [ IP133.itactics.com ]
IP131.itactics.com-stonith (stonith:external/safe/ipmi): Started
IP133.itactics.com FAILED
Failed actions:
IP131.itactics.com-stonith_start_0 (node=IP133.itactics.com, call=8,
rc=1, status=complete): unknown error
============
On doing crm_resource -L :
===============================
Master/Slave Set: ms-drbd
Stopped: [ drbd0:0 drbd0:1 ]
IP131.itactics.com-stonith (stonith:external/safe/ipmi) Started
IP133.itactics.com-stonith (stonith:external/safe/ipmi) Stopped
Resource Group: svcs_grp
fs0 (ocf::heartbeat:Filesystem) Stopped
safe_svcs (ocf::itactics:safe) Stopped
vip (ocf::heartbeat:IPaddr2) Stopped
=======================================
There is disconnect between them as one is the opposite of the other for
IP131.itactics.com-stonith.
We are using ipmi fencing devices.
In addition to that when we did
ps -ef | grep heartbeat we see the following:
root 4185 1 0 15:34 ? 00:00:00
/usr/lib64/heartbeat/stonithd
82 4186 1 0 15:34 ? 00:00:00 /usr/lib64/heartbeat/cib
root 4187 1 0 15:34 ? 00:00:00 /usr/lib64/heartbeat/lrmd
82 4188 1 0 15:34 ? 00:00:00 /usr/lib64/heartbeat/attrd
82 4189 1 0 15:34 ? 00:00:00 /usr/lib64/heartbeat/pengine
82 4190 1 0 15:34 ? 00:00:00 /usr/lib64/heartbeat/crmd
root 7838 4187 0 16:10 ? 00:00:00 /usr/lib64/heartbeat/lrmd
root 7839 4185 0 16:10 ? 00:00:00
/usr/lib64/heartbeat/stonithd
parent-child lrmd and stonithd.
Our config looks like this:
=============================================
node IP131.itactics.com \
attributes standby="off"
node IP133.itactics.com \
attributes standby="off"
primitive IP131.itactics.com-stonith stonith:external/safe/ipmi \
op monitor interval="20s" timeout="3min" \
params target_role="started" hostname="IP131.itactics.com"
ipaddr="192.168.1.2"
primitive IP133.itactics.com-stonith stonith:external/safe/ipmi \
op monitor interval="20s" timeout="2min" \
params target_role="started" hostname="IP133.itactics.com"
ipaddr="192.168.1.3"
primitive drbd0 ocf:linbit:drbd \
params drbd_resource="var_nsm" \
op monitor interval="59s" role="Master" timeout="10s" \
op monitor interval="60s" role="Slave" timeout="10s"
primitive fs0 ocf:heartbeat:Filesystem \
meta target_role="stopped" \
params fstype="xfs" directory="/var/nsm" device="/dev/drbd1"
primitive safe_svcs ocf:itactics:safe \
op start interval="0" timeout="3min" \
op stop interval="0" timeout="3min" \
op monitor interval="20s" timeout="30min" \
params target-role="Started" is-managed="true"
primitive vip ocf:heartbeat:IPaddr2 \
op monitor interval="20s" timeout="1min" \
params ip="172.30.0.139"
group svcs_grp fs0 safe_svcs vip \
meta target_role="started"
ms ms-drbd drbd0 \
meta clone_max="2" clone-node-max="1" notify="yes"
globally-unique="false" master-max="1" master-node-max="1"
target-role="started"
location IP131.itactics.com-stonith-placement IP131.itactics.com-stonith \
rule $id="ri-IP131.itactics.com-stonith-placement-1" -inf: #uname eq
IP131.itactics.com
location IP133.itactics.com-stonith-placement IP133.itactics.com-stonith \
rule $id="ri-IP133.itactics.com-stonith-placement-1" -inf: #uname eq
IP133.itactics.com
location drbd-master ms-drbd \
rule $id="ri-drbd-master-1" $role="master" 100: #uname eq
IP131.itactics.com
colocation svcs-grp-on-ms-drbd inf: svcs_grp ms-drbd:Master
order ms-drbd-before-svcs-group inf: ms-drbd:promote svcs_grp:start
property $id="cib-bootstrap-options" \
dc-version="1.0.5-9e9faaab40f3f97e3c0d623e4a4c47ed83fa1601" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
no-quorum-policy="ignore" \
symmetric-cluster="true" \
stonith-action="reboot" \
stonith-enabled="true" \
dc-deadtime="3min"
rsc_defaults $id="rsc_defaults-options" \
resource-stickiness="100"
===============================================
Appreciate the help.
Thanks
Shravan
Below are my logs:
Dec 2 17:23:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-2, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:3:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:23:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:23:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:23:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:23:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:23:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 3 is
now complete
Dec 2 17:23:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:23:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:23:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 39:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:23:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792619-19, seq=32, quorate=0
Dec 2 17:23:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:23:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:23:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:23:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:23:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:23:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:23:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:23:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:23:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:23:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
4: 14 actions in 14 synapses
Dec 2 17:23:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 4
(ref=pe_calc-dc-1259792619-19) derived from /var/lib/pengine/pe-warn-65.bz2
Dec 2 17:23:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:23:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:23:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:23:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:23:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:23:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:23:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:23:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:23:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:24:34 IP133 crm_resource: [30835]: info: Invoked: crm_resource -L
Dec 2 17:24:35 IP133 crm_resource: [30873]: info: Invoked: crm_resource -L
Dec 2 17:24:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-3, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:4:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:24:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:24:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:24:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:24:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:24:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 4 is
now complete
Dec 2 17:24:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:24:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:24:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 40:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:24:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792679-20, seq=32, quorate=0
Dec 2 17:24:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:24:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:24:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:24:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:24:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:24:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:24:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:24:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:24:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:24:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
5: 14 actions in 14 synapses
Dec 2 17:24:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 5
(ref=pe_calc-dc-1259792679-20) derived from /var/lib/pengine/pe-warn-66.bz2
Dec 2 17:24:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:24:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:24:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:24:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:24:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:24:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:24:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:24:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:24:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:24:40 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:24:41 IP133 crm_resource: [30985]: info: Invoked: crm_resource -L
Dec 2 17:24:57 IP133 crm_resource: [31283]: info: Invoked: crm_resource -L
Dec 2 17:25:10 IP133 crm_resource: [31580]: info: Invoked: crm_resource -L
Dec 2 17:25:13 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:25:15 IP133 dhclient: DHCPDISCOVER on eth3 to 255.255.255.255 port
67 interval 17
Dec 2 17:25:18 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:25:22 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:25:25 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:25:31 IP133 dhclient: DHCPDISCOVER on eth3 to 255.255.255.255 port
67 interval 9
Dec 2 17:25:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-4, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:5:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:25:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:25:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:25:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:25:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:25:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 5 is
now complete
Dec 2 17:25:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:25:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:25:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 41:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:25:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792739-21, seq=32, quorate=0
Dec 2 17:25:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:25:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:25:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:25:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:25:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:25:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:25:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:25:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:25:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:25:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
6: 14 actions in 14 synapses
Dec 2 17:25:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 6
(ref=pe_calc-dc-1259792739-21) derived from /var/lib/pengine/pe-warn-67.bz2
Dec 2 17:25:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:25:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:25:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:25:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:25:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:25:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:25:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:25:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:25:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:26:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-5, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:6:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:26:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:26:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:26:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:26:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:26:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 6 is
now complete
Dec 2 17:26:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:26:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:26:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 42:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:26:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792799-22, seq=32, quorate=0
Dec 2 17:26:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:26:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:26:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:26:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:26:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:26:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:26:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:26:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:26:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:26:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
7: 14 actions in 14 synapses
Dec 2 17:26:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 7
(ref=pe_calc-dc-1259792799-22) derived from /var/lib/pengine/pe-warn-68.bz2
Dec 2 17:26:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:26:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:26:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:26:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:26:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:26:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:26:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:26:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:26:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:27:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-6, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:7:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:27:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:27:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:27:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:27:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:27:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 7 is
now complete
Dec 2 17:27:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:27:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:27:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 43:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:27:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792859-23, seq=32, quorate=0
Dec 2 17:27:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:27:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:27:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:27:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:27:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:27:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:27:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:27:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:27:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:27:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
8: 14 actions in 14 synapses
Dec 2 17:27:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 8
(ref=pe_calc-dc-1259792859-23) derived from /var/lib/pengine/pe-warn-69.bz2
Dec 2 17:27:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:27:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:27:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:27:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:27:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:27:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:27:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:27:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:27:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:28:02 IP133 crm_resource: [2480]: info: Invoked: crm_resource -L
Dec 2 17:28:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-7, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:8:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:28:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:28:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:28:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:28:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:28:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 8 is
now complete
Dec 2 17:28:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:28:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:28:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 44:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:28:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792919-24, seq=32, quorate=0
Dec 2 17:28:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:28:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:28:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:28:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:28:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:28:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:28:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:28:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:28:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:28:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
9: 14 actions in 14 synapses
Dec 2 17:28:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 9
(ref=pe_calc-dc-1259792919-24) derived from /var/lib/pengine/pe-warn-70.bz2
Dec 2 17:28:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:28:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:28:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:28:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:28:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:28:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:28:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:28:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:28:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:29:06 IP133 cib: [24299]: info: cib_stats: Processed 95 operations
(2526.00us average, 0% utilization) in the last 10min
Dec 2 17:29:07 IP133 dhclient: DHCPDISCOVER on eth3 to 255.255.255.255 port
67 interval 20
Dec 2 17:29:08 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:29:18 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:29:19 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:29:27 IP133 dhclient: DHCPDISCOVER on eth3 to 255.255.255.255 port
67 interval 11
Dec 2 17:29:35 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:29:35 IP133 dhclient: DHCPREQUEST on eth1 to 172.30.0.8 port 67
Dec 2 17:29:39 IP133 dhclient: No DHCPOFFERS received.
Dec 2 17:29:39 IP133 dhclient: No working leases in persistent database -
sleeping.
Dec 2 17:29:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-8, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:9:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:29:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:29:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:29:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:29:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:29:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 9 is
now complete
Dec 2 17:29:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:29:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:29:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 45:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:29:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259792979-25, seq=32, quorate=0
Dec 2 17:29:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:29:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:29:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:29:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:29:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:29:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:29:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:29:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:29:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:29:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
10: 14 actions in 14 synapses
Dec 2 17:29:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 10
(ref=pe_calc-dc-1259792979-25) derived from /var/lib/pengine/pe-warn-71.bz2
Dec 2 17:29:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:29:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:29:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:29:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:29:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:29:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:29:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:29:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:29:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
Dec 2 17:30:39 IP133 crmd: [24303]: info: tengine_stonith_callback:
call=-9, optype=1, node_name=IP131.itactics.com, result=2, node_list=,
action=37:10:0:077a920a-e26a-4458-aa47-b2668e6e0189
Dec 2 17:30:39 IP133 crmd: [24303]: info: abort_transition_graph:
tengine_stonith_callback:398 - Triggered transition abort (complete=0) :
Stonith failed
Dec 2 17:30:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
priority upgraded from 0 to 1000000
Dec 2 17:30:39 IP133 crmd: [24303]: info: update_abort_priority: Abort
action done superceeded by restart
Dec 2 17:30:39 IP133 crmd: [24303]: info: run_graph:
====================================================
Dec 2 17:30:39 IP133 crmd: [24303]: info: te_graph_trigger: Transition 10
is now complete
Dec 2 17:30:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Dec 2 17:30:39 IP133 crmd: [24303]: info: do_state_transition: All 1
cluster nodes are eligible to run resources.
Dec 2 17:30:39 IP133 crmd: [24303]: info: do_pe_invoke: Query 46:
Requesting the current CIB: S_POLICY_ENGINE
Dec 2 17:30:39 IP133 crmd: [24303]: info: do_pe_invoke_callback: Invoking
the PE: ref=pe_calc-dc-1259793039-26, seq=32, quorate=0
Dec 2 17:30:39 IP133 pengine: [24302]: info: unpack_config: Node scores:
'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Dec 2 17:30:39 IP133 pengine: [24302]: info: determine_online_status: Node
IP133.itactics.com is online
Dec 2 17:30:39 IP133 pengine: [24302]: info: get_failcount:
IP131.itactics.com-stonith has failed 1000000 times on IP133.itactics.com
Dec 2 17:30:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:30:39 IP133 pengine: [24302]: info: master_color: ms-drbd:
Promoted 0 instances of a possible 1 to master
Dec 2 17:30:39 IP133 pengine: [24302]: info: native_merge_weights: fs0:
Rolling back scores from safe_svcs
Dec 2 17:30:39 IP133 pengine: [24302]: info: native_merge_weights:
safe_svcs: Rolling back scores from vip
Dec 2 17:30:39 IP133 pengine: [24302]: info: native_start_constraints:
Ordering drbd0:0_start_0 after IP131.itactics.com recovery
Dec 2 17:30:39 IP133 crmd: [24303]: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Dec 2 17:30:39 IP133 crmd: [24303]: info: unpack_graph: Unpacked transition
11: 14 actions in 14 synapses
Dec 2 17:30:39 IP133 crmd: [24303]: info: do_te_invoke: Processing graph 11
(ref=pe_calc-dc-1259793039-26) derived from /var/lib/pengine/pe-warn-72.bz2
Dec 2 17:30:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 9
fired and confirmed
Dec 2 17:30:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
10 fired and confirmed
Dec 2 17:30:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action
35 fired and confirmed
Dec 2 17:30:39 IP133 crmd: [24303]: info: te_fence_node: Executing reboot
fencing operation (37) on IP131.itactics.com (timeout=60000)
Dec 2 17:30:39 IP133 stonithd: [24298]: info: client tengine [pid: 24303]
requests a STONITH operation RESET on node IP131.itactics.com
Dec 2 17:30:39 IP133 stonithd: [24298]: info: we can't manage
IP131.itactics.com, broadcast request to other nodes
Dec 2 17:30:39 IP133 stonithd: [24298]: info: Broadcasting the message
succeeded: require others to stonith node IP131.itactics.com.
Dec 2 17:30:39 IP133 crmd: [24303]: info: te_pseudo_action: Pseudo action 7
fired and confirmed
Dec 2 17:30:39 IP133 pengine: [24302]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run "crm_verify
-L" to identify issues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20091202/dac70275/attachment-0001.html>
More information about the Pacemaker
mailing list