[ClusterLabs] Antw: Re: reproducible split brain

[Ken Gaillot]

On 03/17/2016 05:10 PM, Christopher Harvey wrote:
> If I ignore pacemaker's existence, and just run corosync, corosync
> disagrees about node membership in the situation presented in the first
> email. While it's true that stonith just happens to quickly correct the
> situation after it occurs it still smells like a bug in the case where
> corosync in used in isolation. Corosync is after all a membership and
> total ordering protocol, and the nodes in the cluster are unable to
> agree on membership.
> 
> The Totem protocol specifies a ring_id in the token passed in a ring.
> Since all of the 3 nodes but one have formed a new ring with a new id
> how is it that the single node can survive in a ring with no other
> members passing a token with the old ring_id?
> 
> Are there network failure situations that can fool the Totem membership
> protocol or is this an implementation problem? I don't see how it could
> not be one or the other, and it's bad either way.

Neither, really. In a split brain situation, there simply is not enough
information for any protocol or implementation to reliably decide what
to do. That's what fencing is meant to solve -- it provides the
information that certain nodes are definitely not active.

There's no way for either side of the split to know whether the opposite
side is down, or merely unable to communicate properly. If the latter,
it's possible that they are still accessing shared resources, which
without proper communication, can lead to serious problems (e.g. data
corruption of a shared volume).