[Pacemaker] how do I avoid infinite reboot cycles by fencing just the offline node?
Oliver Heinz
oheinz at fbihome.de
Mon Jun 14 16:29:59 UTC 2010
Am Montag, 14. Juni 2010, um 16:43:54 schrieb Dejan Muhamedagic:
> Hi,
>
> On Mon, Jun 14, 2010 at 02:26:57PM +0200, Oliver Heinz wrote:
> > I configured a sbd fencing device on the shared storage to prevent data
> > corruption. It works basically, but when I pull the network plugs on one
> > node to simulate a failure one of the nodes is fenced (not necessarily
> > the one that was unplugged). After the fenced node reboots it fences the
> > other node, this goes on and on...
>
> The networking is still down between the nodes? If so, then this
> is expected.
>
> > I configured pingd and location so that the resources on the shared
> > device are not started on the node that is without network connectivity,
> > but still this node fences the other node.
>
> Yes. With pingd you can influence the resource placement, but it
> can't fix split brain.
>
> > What I would like to achive is that in case of a network problem on a
> > node this node is fenced (and not some randomly chosen node) and that
> > after a reboot this node just sits there waiting for the network to come
> > up again (and not fencing other nodes). Once the network comes up, this
> > node could automatically join the cluster again.
> >
> > Is this possible?
>
> No. You need to make your network connectivity between the nodes
> redundant.
It will be, but I'm testing the worst case scenario. I once had split brain
because I plugged in a firewire device and the kernel oops made it to block any
i/o (network and even the dumb serial line I had for redundancy) for longer
than my configured dead time.
> Split brain is bad news. The cluster will try best to
> deal with it, but, as you could see, it won't always please the
> users.
>
> > Or do I have to disable the cluster stack on bootup, to
> > sort things out manually before joining the cluster.
>
> I think that that's a good idea.
I really like the idea of having the other node sitting there waiting for
network recovery and integrating seamlessly once network is up.
I guess pacemaker fences the other node after it becomes DC, rigth? So what I
would probably want is some rules that prevent this node from becoming DC and
starting to do things. Something like "hmm, I can't reach my gateway I should
keep looking for a DC an not elect myself."
Meanwhile I'll add some checks to the init-Script "if we can't reach the
gateway and the other node we'd better not start the cluster stack"
>
> Oh, and don't make sbd a clone, it doesn't like parallel
> operations to a device.
Thanks for clarification. After reading this
http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg03851.html
I thought it wouldn't hurt either.
Thanks,
Oliver
>
> Thanks,
>
> Dejan
>
> > Can s.o. please point me to the right direction? Maybe I'm just
> > overlooking the obviuos?
> >
> > TIA,
> > Oliver
> >
> > node $id="00b61c9a-22c6-4689-9930-1fd65d5729fa" server-d \
> >
> > attributes standby="off"
> >
> > node $id="0d11e934-91b9-400d-9820-feb2f5895b55" server-c
> > primitive resDATA ocf:heartbeat:LVM \
> >
> > params volgrpname="data"
> >
> > primitive resDataC ocf:heartbeat:Filesystem \
> >
> > params device="/dev/mapper/data-C" directory="/srv/data/C"
> >
> > fstype="ext4" \
> >
> > meta is-managed="true"
> >
> > primitive resDataD ocf:heartbeat:Filesystem \
> >
> > params device="/dev/mapper/data-D" directory="/srv/data/D"
> >
> > fstype="ext4"
> > primitive resPingGateway ocf:pacemaker:pingd \
> >
> > params host_list="gateway"
> >
> > primitive resSBD stonith:external/sbd \
> >
> > params
> > sbd_device="/dev/mapper/3600c0ff000d8d78802faa14b01000000-part1"
> >
> > primitive resVserverTestFramelos ocf:heartbeat:VServer \
> >
> > params vserver="test-framelos" \
> > meta is-managed="true"
> >
> > group grpVserverC resDataC resVserverTestFramelos \
> >
> > meta target-role="Started"
> >
> > group grpVserverD resDataD \
> >
> > meta target-role="Started"
> >
> > clone cloneDATA resDATA
> > clone clonePingGateway resPingGateway \
> >
> > meta target-role="Started"
> >
> > clone cloneSBD resSBD
> > location cli-prefer-grpVserverC grpVserverC \
> >
> > rule $id="cli-prefer-rule-grpVserverC" inf: #uname eq server-c
> >
> > location cli-prefer-grpVserverD grpVserverD \
> >
> > rule $id="cli-prefer-rule-grpVserverD" inf: #uname eq server-d
> > and
> >
> > #uname eq server-d
> > location cli-prefer-resVserverTestFramelos resVserverTestFramelos \
> >
> > rule $id="cli-prefer-rule-resVserverTestFramelos" inf: #uname eq
> >
> > server-c
> > location locPingVserverC grpVserverC \
> >
> > rule $id="locPingVserverC-rule" -inf: not_defined pingd or pingd
> > lte 0
> >
> > location locPingVserverD grpVserverD \
> >
> > rule $id="locPingVserverD-rule" -inf: not_defined pingd or pingd
> > lte 0
> >
> > order ordDataC inf: cloneDATA grpVserverC
> > order ordDataD inf: cloneDATA grpVserverD
> > property $id="cib-bootstrap-options" \
> >
> > dc-version="1.0.8-f2ca9dd92b1d+ sid tip" \
> > cluster-infrastructure="Heartbeat" \
> > expected-quorum-votes="2" \
> > no-quorum-policy="ignore" \
> > stonith-enabled="true" \
> > default-resource-stickiness="INFINITY" \
> > last-lrm-refresh="1276514035"
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs:
> > http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemake
> > r
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
More information about the Pacemaker
mailing list