[Pacemaker] Problems with SBD fencing

Andreas Mock andreas.mock at web.de
Tue Aug 6 05:22:56 EDT 2013 

Hi Dejan,

can you explain how the SDB agent works, when this resource
is running on exactly that node which has to be stonithed?

Thank you in advance.

Best regards
Andreas Mock


-----Ursprüngliche Nachricht-----
Von: Dejan Muhamedagic [mailto:dejanmm at fastmail.fm] 
Gesendet: Dienstag, 6. August 2013 11:15
An: The Pacemaker cluster resource manager
Betreff: Re: [Pacemaker] Problems with SBD fencing

Hi,

On Thu, Aug 01, 2013 at 07:58:55PM +0200, Jan Christian Kaldestad wrote:
> Thanks for the explanation. But I'm quite confused about the SBD stonith
> resource configuration, as the SBD fencing wiki clearly states:
> "The sbd agent does not need to and should not be cloned. If all of your
> nodes run SBD, as is most likely, not even a monitor action provides a
real
> benefit, since the daemon would suicide the node if there was a problem. "
> 
> and also this thread
>
http://oss.clusterlabs.org/pipermail/pacemaker/2012-March/013507.htmlmention
> that there should be only one SBD resource configured.
> 
> Can someone please clarify? Should I configure 2 separate SBD resources,
> one for each cluster node?

No. One sbd resource is sufficient.

Thanks,

Dejan

> 
> -- 
> Best regards
> Jan
> 
> 
> On Thu, Aug 1, 2013 at 6:47 PM, Andreas Mock <andreas.mock at web.de> wrote:
> 
> > Hi Jan,****
> >
> > ** **
> >
> > first of all I don't know the SBD-Fencing-Infrastructure (just read
the***
> > *
> >
> > article linked by you). But as far as I understand the "normal"
fencing***
> > *
> >
> > (initiated on behalf of pacemaker) is done in the following way.****
> >
> > ** **
> >
> > SBD fencing resoure (agent) is writing a request for self-stonithing
into*
> > ***
> >
> > one or more SBD partitions where the SBD-daemon is listening and
hopefully
> > ****
> >
> > reacting on.****
> >
> > So, I'm pretty sure (without knowing) that you have to configure the****
> >
> > stonith agent in a way that pacemaker knows howto talk to the stonith
agent
> > ****
> >
> > to kill a certain cluster node.****
> >
> > What is the problem in you scenario: The agent which should be
contacted**
> > **
> >
> > to stonith the node2 is/was running on node2 and can't be connected
> > anymore.****
> >
> > ** **
> >
> > Because of that stonith agent configuration is most of the times done
the*
> > ***
> >
> > following way in a two node cluster:****
> >
> > On every node runs a stonith agent. The stonith agent is configured
to****
> >
> > stonith the OTHER node. You have to be sure that this is technically
****
> >
> > always possible.****
> >
> > This can be achieved with resource clones or - which is IMHO simpler -
in
> > ****
> >
> > a 2-node-environment with two stonith resources and a negative
colocation*
> > ***
> >
> > constraint.****
> >
> > ** **
> >
> > As far as I know there is also a self-stonith-safty-belt implemented****
> >
> > in a way that a stonith agent on a node to be shot is never
contacted.****
> >
> > (Do I remember correct?)****
> >
> > ** **
> >
> > I'm sure this may solve your problem.****
> >
> > ** **
> >
> > Best regards****
> >
> > Andreas Mock****
> >
> > ** **
> >
> > ** **
> >
> > *Von:* Jan Christian Kaldestad [mailto:janck76 at gmail.com]
> > *Gesendet:* Donnerstag, 1. August 2013 15:46
> > *An:* pacemaker at oss.clusterlabs.org
> > *Betreff:* [Pacemaker] Problems with SBD fencing****
> >
> > ** **
> >
> > Hi,****
> >
> >
> > I am evaluating the SLES HA Extension 11 SP3 product. The cluster
> > consists of 2-nodes (active/passive), using SBD stonith resource on a
> > shared SAN disk. Configuration according to
> > http://www.linux-ha.org/wiki/SBD_Fencing****
> >
> > The SBD daemon is running on both nodes, and the stontih resource
(defined
> > as primitive) is running on one node only.
> > There is also a monitor operation for the stonith resource
> > (interval=36000, timeout=20)****
> >
> > I am having some problems getting failover/fencing to work as expected
in
> > the following scenario:
> > - Node 1 is running the resources that I created (except stonith)
> > - Node 2 is running the stonith resource
> > - Disconnect Node 2 from the network by bringing the interface down
> > - Node 2 status changes to UNCLEAN (offline), but the stonith resource
> > does not switch over to Node 1 and Node 2 does not reboot as I would
expect.
> > - Checking the logs on Node 1, I notice the following:
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node:
Node
> > slesha1n2i-u will be fenced because the node is no longer part of the
> > cluster
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
> > determine_online_status: Node slesha1n2i-u is unclean
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action:
> > Action stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6:
Scheduling
> > Node slesha1n2i-u for STONITH
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions: Move
> > stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
> >  ...
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: te_fence_node:
> > Executing reboot fencing operation (24) on slesha1n2i-u (timeout=60000)
> >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice:
handle_request:
> > Client crmd.8916.3144546f wants to fence (reboot) 'slesha1n2i-u' with
> > device '(any)'
> >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice:
> > initiate_remote_stonith_op: Initiating remote operation reboot for
> > slesha1n2i-u: 8c00ff7b-2986-4b2a-8b4a-760e8346349b (0)
> >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:    error:
remote_op_done:
> > Operation reboot of slesha1n2i-u by slesha1n1i-u for
> > crmd.8916 at slesha1n1i-u.8c00ff7b: No route to host
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > tengine_stonith_callback: Stonith operation
> > 3/24:3:0:8a0f32b2-f91c-4cdf-9cee-1ba9b6e187ab: No route to host (-113)
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > tengine_stonith_callback: Stonith operation 3 for slesha1n2i-u failed
(No
> > route to host): aborting transition.
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > tengine_stonith_notify: Peer slesha1n2i-u was not terminated
> > (st_notify_fence) by slesha1n1i-u for slesha1n1i-u: No route to host
> > (ref=8c00ff7b-2986-4b2a-8b4a-760e8346349b) by client crmd.8916
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: run_graph:
Transition
> > 3 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0,
> > Source=/var/lib/pacemaker/pengine/pe-warn-15.bz2): Stopped
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: unpack_config: On
> > loss of CCM Quorum: Ignore
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node:
Node
> > slesha1n2i-u will be fenced because the node is no longer part of the
> > cluster
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
> > determine_online_status: Node slesha1n2i-u is unclean
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action:
> > Action stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6:
Scheduling
> > Node slesha1n2i-u for STONITH
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions: Move
> > stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
> >  ...
> >  Aug  1 12:00:02 slesha1n1i-u crmd[8916]:   notice:
too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (11), giving up
> >  ****
> >
> >  ****
> >
> >
> > - Then I bring Node 1 online again and start the cluster service,
checking
> > logs:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM
CONFIGURATION
> > CHANGE
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New
Configuration:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
ip(x.x.x.x)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Joined:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_peer_update: Transitional membership event on ring 376: memb=1,
new=0,
> > lost=0
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: memb: slesha1n1i-u 168824371
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM
CONFIGURATION
> > CHANGE
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New
Configuration:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
ip(x.x.x.x)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
ip(y.y.y.y)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Joined:
> >  Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice: ais_dispatch_message:
> > Membership 376: quorum acquired
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0)
ip(y.y.y.y)
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
ais_dispatch_message:
> > Membership 376: quorum acquired
> >  Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice:
crm_update_peer_state:
> > crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now member
> > (was lost)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_peer_update: Stable membership event on ring 376: memb=2, new=1,
lost=0
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node 168824372/slesha1n2i-u is now: member
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: NEW:  slesha1n2i-u 168824372
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
crm_update_peer_state:
> > crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now member
> > (was lost)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: MEMB: slesha1n1i-u 168824371
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
peer_update_callback:
> > Node return implies stonith of slesha1n2i-u (action 24) completed
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: MEMB: slesha1n2i-u 168824372
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [TOTEM ] A processor
joined
> > or left the membership and a new membership was formed.
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: 0x69f2f0 Node 168824372 (slesha1n2i-u) born on: 376
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: crm_update_quorum:
> > Updating quorum status to true (call=119)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CPG   ] chosen downlist:
> > sender r(0) ip(x.x.x.x) ; members(old:1 left:0)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [MAIN  ] Completed
service
> > synchronization, ready to provide service.
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice:
too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:31:14 slesha1n1i-u mgmtd: [8917]: info: CIB query: cib
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node slesha1n2i-u now has process list:
> > 00000000000000000000000000151302 (1381122)
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node slesha1n2i-u now has process list:
> > 00000000000000000000000000141302 (1315586)
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node slesha1n2i-u now has process list:
> > 00000000000000000000000000101302 (1053442)
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: do_state_transition:
> > State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
> > cause=C_HA_MESSAGE origin=route_message ]
> >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice:
too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice:
too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up****
> >
> > - Cluster status changes to Online for both nodes, but the stonith
> > resource won't start on any of the nodes.
> > - Trying to start the resource manually, but no success.
> > - Trying to restart the corosync process on Node 1 (rcopenais restart),
> > but it hangs forever. Checking logs:
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [SERV  ] Unloading all
> > Corosync service engines.
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: Shuting down Pacemaker
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > stop_child: Sent -15 to mgmtd: [8917]
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: mgmtd is shutting
down
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm:
client_id=1
> > cib_name=live
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm:
client_id=2
> > cib_name=live
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: debug: [mgmtd] stopped
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: mgmtd confirmed stopped
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > stop_child: Sent -15 to crmd: [8916]
> >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice: crm_shutdown:
> > Requesting shutdown, upper limit is 1200000ms
> >  Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice:
attrd_trigger_update:
> > Sending flush op to all hosts for: shutdown (1375353728)
> >  Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice:
attrd_perform_update:
> > Sent update 22: shutdown=1375353728
> >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice:
too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:  warning: do_log: FSA: Input
> > I_TE_SUCCESS from abort_transition_graph() received in state
S_POLICY_ENGINE
> >  Aug  1 12:42:38 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to terminate...
> >  Aug  1 12:43:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to
terminate...***
> > *
> >
> > ...****
> >
> > - Finally I kill the corosync process on Node 1 (killall -9 corosync),
> > then corsoync restarts.
> > - Checking status. All resources are up and running on Node 1, and the
> > stonith resource is running on Node 2 again.****
> >
> >  ****
> >
> > I have tested the same scenario several times. Sometimes the fencing
> > mechaism works as expected, but other times the stonith resource is not
> > transferred to Node 1 - as described here. So I need some assistance to
> > overcome this problem....****
> >
> >  ****
> >
> > --
> > Best regards
> > Jan ****
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> >
> 
> 
> -- 
> mvh
> Jan Christian

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list