[Pacemaker] Alternative communication engine to corosync (etcd/consul/zookeeper/doozerd)

[Andrew Beekhof]

On 21 Jun 2014, at 1:32 am, Patrick Hemmer <pacemaker at feystorm.net> wrote:

> From: Andrew Beekhof <andrew at beekhof.net>
> Sent: 2014-06-20 04:48:25 EDT
> To: The Pacemaker cluster resource manager <pacemaker at oss.clusterlabs.org>
> Subject: Re: [Pacemaker] Alternative communication engine to corosync (etcd/consul/zookeeper/doozerd)
> 
>> On 20 Jun 2014, at 2:14 pm, Patrick Hemmer <pacemaker at feystorm.net>
>>  wrote:
>> 
>> 
>>> After the demise of the old heartbeat service, and the switch to corosync as the primary (sole) method of communication between nodes,
>>> 
>> heartbeat is still supported as a messaging/membership layer
>> 
>> 
>>> has there ever been any consideration into using services such as etcd, consul, zookeeper, or doozerd?
>>> 
>> I think most of these didn't exist at the time.
>> Do any provide membership too?  What about message ordering?
>> 
> Yes. N/A (see below on the CPG vs KVS).
>> 
>>> These alternative communication engines offer some stuff that corosync doesn't. One such item etcd & consul offer is dynamic cluster scaling capabilities (nodes can very easily join and leave the cluster). When working in cloud computing, this feature becomes very important. Pcs is somewhat helpful in this regard but it's still nowhere near as capable (plus corosync doesn't have downscaling finished).
>>> However one critical difference between these services and corosync is that they are mainly key/value stores, and don't have something like Corosync's CPG.
>>> 
>> Yes, a rather critical difference.
>> 
>> 
>>> Though you could probably implement something looking like CPG using a keyvalue store,
>>> 
>> I rather doubt it.  k/v stores and CPG are not very alike from where I'm sitting.
> No, they are not alike, but you could implement something looking like CPG.

IF someone else implemented such a thing I would be happy to look at including support for it.
But knowing what I do about the "joy" that went into writing CPG, there is no way that someone is going to be me.

> When a key is created, that's a CPG message. They support atomic operations (collision checking), so no 2 nodes could update a key at the same time. So while it's a different underlying system, the end result is the same, ordered messages.
>>> I think pacemaker might be able to use a key/value store natively.
> But I wouldn't even bother with hacking a KVS into something like CPG if it's not needed. I would do it such that the CIB is stored as keys and values natively. I would even think this is more efficient. I'm not sure how the CIB is transmitted between nodes, but I think it easiest to just set a single key when you want to update something like a resource's last-rc-change value.

CPG is used for a lot more that just keeping the CIB in sync.
I did look at some other KVP stores late last year for the CIB, but I was able to get O(2) speedup without it.

> 
>>> 
>>> So, has this ever been considered? How heavily tied is pacemaker to the corosync API? Could that be abstracted out enough to where different communication engines could be implemented?
>>> 
>> It's already abstracted to the point where it supports 4 different messaging/membership options:
>> - heartbeat
>> - corosync/pacemaker plugin
>> - cman
>> - corosync 2
>> 
> But aside from heartbeat, they all use corosync underneath.

Technically yes, but they all expose very different APIs so they don't look at all alike to pacemaker.

> I wasn't aware heartbeat was still supported. I assumed it was dropped. Bad assumption :-)
> 
>> 
>>> 
>>> -Patrick
>>> _______________________________________________
>>> Pacemaker mailing list: 
>>> Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> 
>>> 
>>> Project Home: 
>>> http://www.clusterlabs.org
>>> 
>>> Getting started: 
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> 
>>> Bugs: 
>>> http://bugs.clusterlabs.org
>> 
>> 
>> _______________________________________________
>> Pacemaker mailing list: 
>> Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> 
>> Project Home: 
>> http://www.clusterlabs.org
>> 
>> Getting started: 
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> 
>> Bugs: 
>> http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20140622/edcb78f4/attachment-0001.sig>