[ClusterLabs] Node Fencing and STONITH in Amazon Web Services

[Enno Gröper]

Hi,

Am 26.08.2016 um 22:32 schrieb Jason A Ramsey:
> No users
> would be connecting to the severed instances, but background and system
> tasks would proceed as normal, potentially writing new data to the
> databases making rejoining the nodes to the cluster a little bit tricky
> to say the least, especially if the severed side’s network comes back up
> and both systems come to realize that they’re not consistent.
I think you forgot a potential routing problem. Network is not simply
on/off.
It could be, that you have a split brain scenario, where your 2 nodes
don't see each other. But it is perfectly possible, that at the same
time both nodes are seen by / interacting with some of your users
(depending on their location/routing). So it aren't just some background
tasks potentially writing data to the databases. It may be real user data.

I'm interested in this topic as this is a general cloud problem: To my
knowledge you simply don't have any out-of-band messaging channel
between your nodes to avoid split brain or make STONITH possible.
At least in OpenStack (this is what I know), everything running on the
node (vm) needs to go through client networking, which could be
malfunctioning. Even if it is, in theory, possible to issue API calls to
shutdown the other node. These calls would still need to go through the
same messaging channel (client network).

A solution could be to use DBaaS (reliable database provided by cloud
service provider). Don't know if any csp provides database replication
across different sites.
I simply don't see a way to reliably solve this using Pacemaker (without
out-of-band messaging channel and/or reliable STONITH).
Imho for DBaaS you need to look carefully at the SLA / specs to see, if
your DBaaS really provides, what you want.

My 2 cents
Enno

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5046 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160829/ae12281b/attachment-0004.p7s>