[Pacemaker] Cluster with DRBD : split brain
Andreas Kurz
andreas at hastexo.com
Thu Apr 5 15:28:18 UTC 2012
On 04/04/2012 03:40 PM, Hugo Deprez wrote:
> Hello,
>
> thanks for the information.
> I was looking at this page
> http://www.drbd.org/users-guide/s-pacemaker-fencing.html
> I did specify the following handlers :
> handlers {
> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
>
> }
>
> I disconnected the network cable between the clusters, corosync and drbd
> uses this link.
>
> I was able to see that the fence script added a constraint :
>
> location drbd-fence-by-handler-ms-drbd-supervision ms-drbd-supervision \
> rule $id="drbd-fence-by-handler-rule-ms-drbd-supervision"
> $role="Master" -inf: #uname ne host
As expected, now Pacemaker won't try to promote that DRBD resource on
any node but "host"
> But this made :
>
> :StandAlone ro:Secondary/Unknown ds:UpToDate/Outdated on drbd.
also expected
>
> I don't really understand what I should be expected from those handlers ?
> When cleaning up the errors, I shoudl delete the constraint right ?
>
The constraint is cleared automatically, once the resync is finished -->
after-resync-target handler ... after your did the cleanup and
reconnected the resources.
Regards,
Andreas
--
Need help with Pacemaker?
http://www.hastexo.com/now
> Regards,
>
> Hugo
>
> On 26 July 2011 19:27, Digimer <linux at alteeve.com
> <mailto:linux at alteeve.com>> wrote:
>
> On 07/26/2011 11:43 AM, Lars Ellenberg wrote:
> > On Wed, Jul 20, 2011 at 11:36:25AM -0400, Digimer wrote:
> >> On 07/20/2011 11:24 AM, Hugo Deprez wrote:
> >>> Hello Andrew,
> >>>
> >>> in fact DRBD was in standalone mode but the cluster was working :
> >>>
> >>> Here is the syslog of the drbd's split brain :
> >>>
> >>> Jul 15 08:45:34 node1 kernel: [1536023.052245] block drbd0:
> Handshake
> >>> successful: Agreed network protocol version 91
> >>> Jul 15 08:45:34 node1 kernel: [1536023.052267] block drbd0: conn(
> >>> WFConnection -> WFReportParams )
> >>> Jul 15 08:45:34 node1 kernel: [1536023.066677] block drbd0: Starting
> >>> asender thread (from drbd0_receiver [23281])
> >>> Jul 15 08:45:34 node1 kernel: [1536023.066863] block drbd0:
> >>> data-integrity-alg: <not-used>
> >>> Jul 15 08:45:34 node1 kernel: [1536023.079182] block drbd0:
> >>> drbd_sync_handshake:
> >>> Jul 15 08:45:34 node1 kernel: [1536023.079190] block drbd0: self
> >>> BBA9B794EDB65CDF:9E8FB52F896EF383:C5FE44742558F9E1:1F9E06135B8E296F
> >>> bits:75338 flags:0
> >>> Jul 15 08:45:34 node1 kernel: [1536023.079196] block drbd0: peer
> >>> 8343B5F30B2BF674:9E8FB52F896EF382:C5FE44742558F9E0:1F9E06135B8E296F
> >>> bits:769 flags:0
> >>> Jul 15 08:45:34 node1 kernel: [1536023.079200] block drbd0:
> >>> uuid_compare()=100 by rule 90
> >>> Jul 15 08:45:34 node1 kernel: [1536023.079203] block drbd0:
> Split-Brain
> >>> detected, dropping connection!
> >>> Jul 15 08:45:34 node1 kernel: [1536023.079439] block drbd0: helper
> >>> command: /sbin/drbdadm split-brain minor-0
> >>> Jul 15 08:45:34 node1 kernel: [1536023.083955] block drbd0: meta
> >>> connection shut down by peer.
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084163] block drbd0: conn(
> >>> WFReportParams -> NetworkFailure )
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084173] block drbd0: asender
> >>> terminated
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084176] block drbd0:
> Terminating
> >>> asender thread
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084406] block drbd0: helper
> >>> command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084420] block drbd0: conn(
> >>> NetworkFailure -> Disconnecting )
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084430] block drbd0: error
> >>> receiving ReportState, l: 4!
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084789] block drbd0:
> Connection
> >>> closed
> >>> Jul 15 08:45:34 node1 kernel: [1536023.084813] block drbd0: conn(
> >>> Disconnecting -> StandAlone )
> >>> Jul 15 08:45:34 node1 kernel: [1536023.086345] block drbd0: receiver
> >>> terminated
> >>> Jul 15 08:45:34 node1 kernel: [1536023.086349] block drbd0:
> Terminating
> >>> receiver thread
> >>
> >> This was a DRBD split-brain, not a pacemaker split. I think that
> might
> >> have been the source of confusion.
> >>
> >> The split brain occurs when both DRBD nodes lose contact with one
> >> another and then proceed as StandAlone/Primary/UpToDate. To avoid
> this,
> >> configure fencing (stonith) in Pacemaker, then use
> 'crm-fence-peer.sh'
> >> in drbd.conf;
> >>
> >> ===
> >> disk {
> >> fencing resource-and-stonith;
> >> }
> >>
> >> handlers {
> >> outdate-peer "/path/to/crm-fence-peer.sh";
> >> }
> >> ===
> >
> > Thanks, that is basically right.
> > Let me fill in some details, though:
> >
> >> This will tell DRBD to block (resource) and fence (stonith). DRBD
> will
> >
> > drbd fencing options are "fencing resource-only",
> > and "fencing resource-and-stonith".
> >
> > "resource-only" does *not* block IO while the fencing handler runs.
> >
> > "resource-and-stonith" does block IO.
>
> Ahhh, that's why I was confused. I thought the 'resource' meant the same
> thing in both cases, but had only read the 'resource-and-stonith'
> section.
>
> >> not resume IO until either the fence script exits with a success, or
> >> until an admit types 'drbdadm resume-io <res>'.
> >
> >
> >> The CRM script simply calls pacemaker and asks it to fence the other
> >> node.
> >
> > No. It tries to place a constraint forcing the Master role off of any
> > node but the one with the good data.
>
> Ok, I thought it was akin to the 'obliterate-peer.sh' script, which
> calls 'fence_node'... I made an assumption, which was not correct.
>
> >> When a node has actually failed, then the lost no is fenced. If
> >> both nodes are up but disconnected, as you had, then only the fastest
> >> node will succeed in calling the fence, and the slower node will be
> >> fenced before it can call a fence.
> >
> > "fenced" may be "restricted from being/becoming Master" by that
> fencing
> > constraint. Or, if pacemaker decided to do so, actually "shot" by some
> > node level fencing agent (stonith).
> >
> > All that resource-level fencing by placing some constraint stuff
> > obviously only works as long as the cluster communication is still up.
> > It not only the drbd replication link had issues, but the cluster
> > communication was down as well, it becomes a bit more complex.
>
> Thanks for the clarity. Today I learned. :)
>
> --
> Digimer
> E-Mail: digimer at alteeve.com <mailto:digimer at alteeve.com>
> Freenode handle: digimer
> Papers and Projects: http://alteeve.com
> Node Assassin: http://nodeassassin.org
> "At what point did we forget that the Space Shuttle was, essentially,
> a program that strapped human beings to an explosion and tried to stab
> through the sky with fire and math?"
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 222 bytes
Desc: OpenPGP digital signature
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20120405/d7b17ea4/attachment-0004.sig>
More information about the Pacemaker
mailing list