[Pacemaker] Problem with dual-PDU fencing node with redundant PSUs

[Digimer]

On 06/27/2013 11:08 AM, Lars Marowsky-Bree wrote:
> On 2013-06-27T10:56:40, Digimer <lists at alteeve.ca> wrote:
> 
>> However, this feels like a really bad solution. It's not uncommon to
>> have two separate power rails feeding either side of the node's PSUs.
>> Particularly in HA environments.
> 
> True. But gating them through the same power switch is *not* a SPoF from
> the cluster's perspective, "just" for the single node (if the power
> switch fails).

Actually, it is. If the single PDU dies, then you lose IPMI based
fencing and PDU fencing, so the cluster can not (safely) recover,
leaving things blocked until a human intervenes.

> On the other hand, each of the two switches/PDUs (and the network
> interconnect to each) becomes a SPoF for *fencing* the node, since you
> need an ACK from both; two PDU approval from both. Basically, that
> doubles the unreliability of the environment. And, if, indeed, you lose
> power to one of the grids, *you can no longer fence* via this
> mechanism.

In my setups, this is not the case. I use two switches (all nodes are
bonded across the two) with IPMI in the first switch and the PDUs in the
second switch.

If a PDU dies, then the IPMI interface is still powered and will work
for fencing.

If a switch dies, then either IPMI or the PDU fencing remains available.

So no single failure will take out fencing.

> Thus, this only makes sense as a fall-back mechanism, obviously. If we
> have both (say, IPMI + dual switch), we actually want to not try them in
> sequence though, but in parallel - to lower recovery time. (Waiting for
> the IPMI network timeout isn't nice.)

I want them in sequence because an IPMI "success" is more trust-worthy
than the PDUs "success" (which says only "yes, the outlets were opened).
You are right, it slows down recovery, but the PDUs should not be needed
except in corner cases, so given the preference to IPMI fencing, I am
willing to incur the extra recovery time in such cases.

> Personally, I've tried to discourage users from building such
> environments. Since most of our customers have something like shared
> storage, I much prefer shared storage based fencing these days.

An alternate philosophy. :)

>> time and I expect many users will run into this problem as they try to
>> migrate to RHEL 7. I see no reason why this can't be properly handled in
>> pacemaker directly.
> 
> Yes, why not, choice is a good thing ;-)

If an established configuration is not supported in the only remaining
HA stack (RHEL 7 won't have cman/rgmanager anymore), then there is no
choice.

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?