[ClusterLabs] reproducible split brain

[Digimer]

On 16/03/16 03:59 PM, Christopher Harvey wrote:
> I am able to create a split brain situation in corosync 1.1.13 using
> iptables in a 3 node cluster.
> 
> I have 3 nodes, vmr-132-3, vmr-132-4, and vmr-132-5
> 
> All nodes are operational and form a 3 node cluster with all nodes are
> members of that ring.
> vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
> vmr-132-4 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
> vmr-132-5 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
> so far so good.
> 
> running the following on vmr-132-4 drops all incoming (but not outgoing)
> packets from vmr-132-3:
> # iptables -I INPUT -s 192.168.132.3 -j DROP
> # iptables -L
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination
> DROP       all  --  192.168.132.3        anywhere
> 
> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination
> 
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination
> 
> vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
> vmr-132-4 ---> Online: [ vmr-132-4 vmr-132-5 ]
> vmr-132-5 ---> Online: [ vmr-132-4 vmr-132-5 ]
> 
> vmr-132-3 thinks everything is normal and continues to provide service,
> vmr-132-4 and 5 form a new ring, achieve quorum and provide the same
> service. Splitting the link between 3 and 4 in both directions isolates
> vmr 3 from the rest of the cluster and everything fails over normally,
> so only a unidirectional failure causes problems.
> 
> I don't have stonith enabled right now, and looking over the
> pacemaker.log file closely to see if 4 and 5 would normally have fenced
> 3, but I didn't see any fencing or stonith logs.
> 
> Would stonith solve this problem, or does this look like a bug?

It should, that is its job.

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?