[ClusterLabs] Antw: Re: reproducible split brain

[Dennis Jacobfeuerborn]

On 18.03.2016 00:50, Digimer wrote:
> On 17/03/16 07:30 PM, Christopher Harvey wrote:
>> On Thu, Mar 17, 2016, at 06:24 PM, Ken Gaillot wrote:
>>> On 03/17/2016 05:10 PM, Christopher Harvey wrote:
>>>> If I ignore pacemaker's existence, and just run corosync, corosync
>>>> disagrees about node membership in the situation presented in the first
>>>> email. While it's true that stonith just happens to quickly correct the
>>>> situation after it occurs it still smells like a bug in the case where
>>>> corosync in used in isolation. Corosync is after all a membership and
>>>> total ordering protocol, and the nodes in the cluster are unable to
>>>> agree on membership.
>>>>
>>>> The Totem protocol specifies a ring_id in the token passed in a ring.
>>>> Since all of the 3 nodes but one have formed a new ring with a new id
>>>> how is it that the single node can survive in a ring with no other
>>>> members passing a token with the old ring_id?
>>>>
>>>> Are there network failure situations that can fool the Totem membership
>>>> protocol or is this an implementation problem? I don't see how it could
>>>> not be one or the other, and it's bad either way.
>>>
>>> Neither, really. In a split brain situation, there simply is not enough
>>> information for any protocol or implementation to reliably decide what
>>> to do. That's what fencing is meant to solve -- it provides the
>>> information that certain nodes are definitely not active.
>>>
>>> There's no way for either side of the split to know whether the opposite
>>> side is down, or merely unable to communicate properly. If the latter,
>>> it's possible that they are still accessing shared resources, which
>>> without proper communication, can lead to serious problems (e.g. data
>>> corruption of a shared volume).
>>
>> The totem protocol is silent on the topic of fencing and resources, much
>> the way TCP is.
>>
>> Please explain to me what needs to be fenced in a cluster without
>> resources where membership and total message ordering are the only
>> concern. If fencing were a requirement for membership and ordering,
>> wouldn't stonith be part of corosync and not pacemaker?
> 
> Corosync is a membership and communication layer (and in v2+, a quorum
> provider). It doesn't care about or manage anything higher up. So it
> doesn't care about fencing itself.
> 
> It simply cares about;
> 
> * Who is in the cluster?
> * How do the members communicate?
> * (v2+) Is there enough members for quorum?
> * Notify resource managers of membership changes (join or loss).
> 
> The resource manager, pacemaker or rgmanager, care about resources, so
> it is what cares about making smart decisions. As Ken pointed out,
> without fencing, it can never tell the difference between no access and
> dead peer.
> 
> This is (again) why fencing is critical.

I think the key issue here is that people think about corosync they
believe there can only be two state for membership (true or false) when
in reality there are three possible states: true, false and unknown.

The problem then is that corosync apparently has no built-in way to deal
with the "unknown" situation and requires guidance from an external
entity for that (in this case pacemakers fencing).

This means that corosync alone simply cannot give you reliable
membership guarantees. I strictly requires external help to be able to
provide that.

Regards,
  Dennis