[Pacemaker] How to avoid dual fencing ?

[Digimer]

Hi 邓尧,

   This question requires a little longer answer.

   The delay that Kristoffer talks about is where you set one node to 
wait a certain number of seconds before fencing the other node. In the 
opposite direction, the other node does not wait to fence, so in the 
case where both nodes try to fence at the same time, the one that pauses 
will lose because the one that did not pause will shut down the slow 
node before it's timer expires. Generally, you pick a node you prefer to 
survive in this case and add 'delay="15"' to it's fence/stonith 
configuration.

   This isn't enough though. Let me explain why in a moment.

   First, consider for example, say you want node 1 to win in these 
cases. When you setup node 1's fence method, which is the instructions 
the rest of the cluster uses to fence node 1, you add 'delay="15"'. For 
node 2, you set no delay at all. Now something breaks the connection 
between the nodes corosync communications, but not fencing (say a stray 
iptables rule or similar). Both nodes see the other as lost and both 
nodes immediately start to fence the peer. Node 1 gathers up the fencing 
details on node 2 and immediately calls the fence. Meanwhile, node 2 
gathers up the details to fence node 1, sees a 15 second delays and 
pauses. Node 1 will win.

   Now the problem.

   Consider that your switch (or stack of switches) between the nodes 
fails. Consider a fence-loop or an errant command rebooting the stack. 
All channels are lost, so neither corosync traffic or fencing works.

   Again, node 1 immediately calls a fence against node 2. This time 
though, the fence fails, so node 1 goes into a loop trying to fence node 
1. At the same time, node 2 starts a fence, pauses 15 seconds, then 
calls the fence against node 1. This also fails, so it goes into a loop 
as well.

   Once the switches recovery, both nodes are sitting there trying to 
fence the other immediately, without delay. So you're back to the 
problem where both nodes could succeed in fencing the other, leaving 
both nodes powered off.

   This scenario is very possible when IPMI fencing is used. Normally, 
if 'acpid' is running on your nodes, they will react to the IPMI BMC 
"pressing the power button" by beginning a graceful shut down. The BMC 
presses and holds the power button, so after four seconds, the node will 
forcibly power off. However, this is more than enough time for the 
slower node to begin shut down of it's peer.

   So to prevent this, disable acpid.

   In most systems, when acpid is _not_ running, the system will 
instantly power off when it receives a power-button event. In this case, 
the time between the faster node pressing the slower node's power button 
and the slow node being disabled is nearly instant. *Far* less likely 
that the slower node will get it's fence call out before dying.

   So in short;

1. Pick a node you prefer to win, add 'delay="15"' to it's fence/stonith 
configuration. Set no delay on the other node.

2. Disable acpid.

   Hope this helps!

digimer

On 24/03/14 10:28 PM, 邓尧 wrote:
> How much time delay difference is enough ? I'm currently setting the
> delays at 5s and 15s, dual fencing does disappear, but I'm sure whether
> the difference is large enough for all situations.
>
>
> On Mon, Mar 24, 2014 at 8:00 PM, Kristoffer Grönlund <kgronlund at suse.com
> <mailto:kgronlund at suse.com>> wrote:
>
>     On Mon, 24 Mar 2014 19:41:16 +0800
>     邓尧 <torshie at gmail.com <mailto:torshie at gmail.com>> wrote:
>
>      > I'm configuring a 2-node cluster on CentOS 6 with corosync +
>      > pacemaker + cman. Everything works well except fencing.
>      >
>      > I'm using IPMI as the fencing device, I know it's not the best
>     fencing
>      > device, but this is the only option I have.
>      > Manual fencing a node (pcs stonith fence <node>) works well: peer
>      > node goes offline, resources migrate as expected. However, if network
>      > traffic between the two nodes is blocked by iptables, both nodes will
>      > try to fence each other, and both would success, which result in both
>      > nodes offline.
>      >
>      > How to avoid such problem ?
>
>     You'll need to have one node take priority, by setting different
>     delays on the two nodes so that one of them will fence before the
>     other (see the "delay" parameter).
>
>     --
>     // Kristoffer Grönlund
>     // kgronlund at suse.com <mailto:kgronlund at suse.com>
>
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>


-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?