[ClusterLabs] Antw: Re: reproducible split brain

[Bogdan Dobrelya]

On 03/17/2016 11:24 PM, Ken Gaillot wrote:
> On 03/17/2016 05:10 PM, Christopher Harvey wrote:
>> If I ignore pacemaker's existence, and just run corosync, corosync
>> disagrees about node membership in the situation presented in the first
>> email. While it's true that stonith just happens to quickly correct the
>> situation after it occurs it still smells like a bug in the case where
>> corosync in used in isolation. Corosync is after all a membership and
>> total ordering protocol, and the nodes in the cluster are unable to
>> agree on membership.
>>
>> The Totem protocol specifies a ring_id in the token passed in a ring.
>> Since all of the 3 nodes but one have formed a new ring with a new id
>> how is it that the single node can survive in a ring with no other
>> members passing a token with the old ring_id?
>>
>> Are there network failure situations that can fool the Totem membership
>> protocol or is this an implementation problem? I don't see how it could
>> not be one or the other, and it's bad either way.
> 
> Neither, really. In a split brain situation, there simply is not enough
> information for any protocol or implementation to reliably decide what
> to do. That's what fencing is meant to solve -- it provides the
> information that certain nodes are definitely not active.
> 
> There's no way for either side of the split to know whether the opposite
> side is down, or merely unable to communicate properly. If the latter,

I completely agree and believe that STONITH resolves the unreliable
failure detection problem of distributed systems in a simple and elegant
way, which is much more safe than timeouts based detection, for example.

I'm also wondering why having been serving production clusters for *so
long*, Pacemaker (BUT not support teams of some Linux distros though!)
stays so loyal allowing to run without STONITH? Why not to deprecate
that dangerous config option and remove it shortly? Let's just make the
pacemaker service refusing to start w/o fencing primitives configured.

And please don't blame me for being a bit radical. First, I've suggested
the deprecation period with BIG RED ALARMS in logs and stdouts. And
second, I have sad experience maintaining HA in a product that relies on
a Pacemaker cluster but refuses to support configuring STONITH out of
box for years(!), choosing another fancy features with high business
values instead. So, may be deprecating and removing STONITH-less mode
would encourage ppl to reconsider things (like a cold shower :-))

> it's possible that they are still accessing shared resources, which
> without proper communication, can lead to serious problems (e.g. data
> corruption of a shared volume).
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando