[Pacemaker] [RFC] Automatic nodelist synchronization between corosync and pacemaker

Vladislav Bogdanov bubble at hoster-ok.com
Tue Mar 12 03:51:09 UTC 2013 

12.03.2013 04:44, Andrew Beekhof wrote:
> On Thu, Mar 7, 2013 at 5:30 PM, Vladislav Bogdanov <bubble at hoster-ok.com> wrote:
>> 07.03.2013 03:37, Andrew Beekhof wrote:
>>> On Thu, Mar 7, 2013 at 2:41 AM, Vladislav Bogdanov <bubble at hoster-ok.com> wrote:
>>>> 06.03.2013 08:35, Andrew Beekhof wrote:
>>>
>>>>> So basically, you want to be able to add/remove nodes from nodelist.*
>>>>> in corosync.conf and have pacemaker automatically add/remove them from
>>>>> itself?
>>>>
>>>> Not corosync.conf, but cmap which is initially (partially) filled with
>>>> values from corosync.conf.
>>>>
>>>>>
>>>>> If corosync.conf gets out of sync (admin error or maybe a node was
>>>>> down when you updated last) they might well get added back - I assume
>>>>> you're ok with that?
>>>>> Because there's no real way to know the difference between "added
>>>>> back" and "not removed from last time".
>>>>
>>>> Sorry, can you please reword?
>>>
>>> When node-A comes up with "node-X" that no-one else has, the cluster
>>> has no way to know if node-X was just added, or if the admin forgot to
>>> remove it on node-A.
>>
>> Exactly that is not problem if node does not appear in CIB until it is
>> seen online.
> 
> But that is at odds with the "read the corosync nodelist" part.

But not with
* Do not add nodes from nodelist to CIB if their join-count in cmap is
zero (but do not touch CIB nodes which exist in a nodelist and have zero
join-count in cmap).


For the rest - I got your position on node deletion. It is not very
common to see so many words from you, so, thank you for interesting
discussion ;)

> 
>> If node-A comes up, then it is just booted, and that means
>> that it didn't see node-X online yet (if it is not actually online of
>> course). And then node-X is not added to CIB.
>>
>>>
>>>>> Or are you planning to never update the on-disk corosync.conf and only
>>>>> modify the in-memory nodelist?
>>>>
>>>> That depends on the actual use case I think.
>>>>
>>>> Hm. Interesting, how corosync behave when new dynamic nodes are added to
>>>> cluster... I mean following: we have static corosync.conf with nodelist
>>>> containing f.e. 3 entries, then we add fourth entry via cmap and boot
>>>> fourth node. What should be in corosync.conf of that node?
>>>
>>> I don't know actually.  Try it and see if it works without the local
>>> node being defined?
>>>
>>>> I believe in
>>>> wont work without that _its_ fourth entry. Ugh. If so, then no fully
>>>> dynamic "elastic" cluster which I was dreaming of is still possible
>>>> because out-of-the-box when using dynamic nodelist.
>>>>
>>>> The only way to have this I see is to have static nodelist in
>>>> corosync.conf with all possible nodes predefined. And never edit it in
>>>> cmap. So, my original point
>>>> * Remove nodes from CIB when they are removed from a nodelist.
>>>> does not fit.
>>>>
>>>> By elastic I mean what was discussed on corosync list when Fabio started
>>>> with votequorum design and what then appeared in votequorum manpage:
>>>> ===========
>>>> allow_downscale: 1
>>>>
>>>> Enables allow downscale (AD) feature (default: 0).
>>>>
>>>> The general behaviour of votequorum is to never decrease expected votes
>>>> or quorum.
>>>>
>>>> When  AD  is  enabled,  both expected votes and quorum are recalculated
>>>> when a node leaves the cluster in a clean state (normal corosync  shut-
>>>> down process) down to configured expected_votes.
>>>
>>> But thats very different to removing the node completely.
>>> You still want to know its in a sane state.
>>
>> Isn't it enough to trust corosync here?
> 
> Absolutely not.
> "Clean" to corosync means "did corosync on that node send me a message
> to say that they planned to exit".
> That implies nothing at all about the state of pacemaker or, more
> importantly, the cluster resources on that machine.
> 
> In addition, "clean" is an internal corosync concept that is not
> reported to clients like pacemaker.
> 
>> Of course if it supplies some event that "node X leaved cluster in a
>> clean state and we lowered expected_votes and quorum.
>>
>> Clean corosync shutdown means that either 'no more corosync clients
>> remain and it was safe to shutdown' or 'corosync has a bug'.
>> Pacemaker is corosync client, and corosync should not stop in a clean
>> state if pacemaker is still running there.
>>
>> And 'pacemaker is not running on node X' means that pacemaker instances
>> on other nodes accepted that. Otherwise node is scheduled to stonith and
>> there is no 'clean' shutdown.
>>
>> Am I correct here?
> 
> Not really, no.
> 
>>>
>>>> Example use case:
>>>>
>>>> 1) N node cluster (where N is any value higher than 3)
>>>> 2) expected_votes set to 3 in corosync.conf
>>>> 3) only 3 nodes are running
>>>> 4) admin requires to increase processing power and adds 10 nodes
>>>> 5) internal expected_votes is automatically set to 13
>>>> 6) minimum expected_votes is 3 (from configuration)
>>>> - up to this point this is standard votequorum behavior -
>>>> 7) once the work is done, admin wants to remove nodes from the cluster
>>>> 8) using an ordered shutdown the admin can reduce the cluster size
>>>>    automatically back to 3, but not below 3, where normal quorum
>>>>    operation will work as usual.
>>>> =============
>>>>
>>>> What I would expect from pacemaker, is to automatically remove nodes
>>>> down to 3 at step 8 (just follow quorum) if AD is enabled AND pacemaker
>>>> is instructed to follow that (with some other cmap switch). And also to
>>>> reduce number of allocated clone instances. Sure, all nodes must have
>>>> equal number of votes (1).
>>>>
>>>> Is it ok for you?
>>>
>>> Not really.
>>> We simply don't have enough information to do the removal.
>>> All we get is "node gone", we have to do a fair bit of work to
>>> calculate if it was clean at the time or not (and clean to corosync
>>> doesn't always imply clean to pacemaker).
>>
>> Please see above.
>> There is always (at least with mcp model) some time frame between
>> pacemaker stop and corosync stop events.
> 
> Relying on this would be a world of hurt.
> I have enough trouble dealing with timing issues to ever consider
> relying on one.
> 
> Nodes routinely reappear in virtualised clusters before stonith
> reports they have been fenced.
> Or peers that show up in the CPG membership 5 minutes before appearing
> in the cman/quorum membership.
> 
>> And pacemaker should accept
>> "node leave" after first one (doesn't it mark node as 'pending' in that
>> state?). And second event (corosync stop) is enough to remove that
>> 'pending' node I think.
>>
>> And, I think there should be some special message from corosync (or
>> better from votequorum)
> 
> Please. Stop. votequorum excluding a node's vote is completely
> different to removing all trace of the node from the cluster.
> One is not a suitable trigger for the other.
> 
>> that 'I lowered quorum and expected_votes
>> accepting clean leave of node X'. Otherwise we'd have problems when two
>> node leave at the same time, but one is clean while other is not. I hope
>> guys who maintain corosync will accept patch with that if it is not
>> possible to get similar message with current implementation.
>>
>>>
>>> So back to the start, why do you need pacemaker to forget about the
>>> other 10 nodes?
>>
>> Mostly aesthetic reasons.
> 
> This is a massive amount of pain and complexity to create for the
> "aesthetic reasons" of (AFAICS) a very uncommon use-case.
> Why not simply set value-- for clone-max as part of the shutdown
> procedure for your extra nodes?
> 
> (The cib actually understands value="value--")
> 
>> I think about using pacemaker as a backend for private (or semi-private)
>> virtualization clouds with varying load. F.e. 10 nodes, 3 of then should
>> run always, remaining 7 are only powered on when peak load is requested
>> (some heavy calculations should be done as fast as possible). Power is
>> not cheap today, so I'd prefer to have that nodes powered off when they
>> are not needed. That peak load is required only 3 days per month.
>> So I want cluster to "breath" accordingly with user needs.
>>
>> I agree that nothing hampers that to be in case right now, and that will
>> work, but f.e. crm_mon output will show all allocated but stopped clone
>> instances, and it would be much harder for admin to quickly find real
>> problems.
>>
>> Please look:
>> * No extra nodes in CIB, no extra clone instances
>> ** Everything is good:
>>
>>  Clone Set: cl-vlan200-if [vlan200-if]
>>      Started: [ v03-b v03-a v03-c ]
>>
>> ** Real problem, cl-vlan200-if is not running on v03-c
>>
>>  Clone Set: cl-vlan200-if [vlan200-if]
>>      Started: [ v03-b v03-a ]
>>      Stopped: [ vlan200-if:2 ]
>>
>> * Extra nodes in CIB, extra clone instances
>> ** Everything is good:
>>
>>  Clone Set: cl-vlan200-if [vlan200-if]
>>      Started: [ v03-b v03-a v03-c ]
>>      Stopped: [ vlan200-if:3 vlan200-if:4 vlan200-if:5 vlan200-if:6
>> vlan200-if:7 ]
>>
>> ** Real problem, cl-vlan200-if is not running on v03-c
>>
>>  Clone Set: cl-vlan200-if [vlan200-if]
>>      Started: [ v03-b v03-a ]
>>      Stopped: [ vlan200-if:2 vlan200-if:3 vlan200-if:4 vlan200-if:5
>> vlan200-if:6 vlan200-if:7 ]
>>
>> Do last two differ much, especially if there are dozens of such lines?
>> I think no.
>>
>> But admin quickly sees problem in the first case. And, that is probably
>> related not only to crm_mon, but other management tools as well, because
>> there is mostly no generic way to distinguish between 'correctly
>> not-running clone instance' and 'accidentally not running instance'
>> without involving extra logic which complicates management tools a lot.
>>
>> Does this make clearer why I what that all to be implemented?
>>
>>> (because everything apart from that should already work).
>>>
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>> That
>>>>>>>> would be OK if number of clone instances does not raise with that...
>>>>>>>
>>>>>>> Why?  If clone-node-max=1, then you'll never have more than the number
>>>>>>> of active nodes - even if clone-max is greater.
>>>>>>
>>>>>> Active (online) or known (existing in a <nodes> section)?
>>>>>> I've seen that as soon as node appears in <nodes> even in offline state,
>>>>>> new clone instance is allocated.
>>>>>
>>>>> $num_known instances will "exist", but only $num_active will be running.
>>>>
>>>> Yep, that's what I say. I see them in crm_mon or 'crm status' and they
>>>> make my life harder ;)
>>>> That remaining instances are "allocated" but not running.
>>>>
>>>> I can agree that this issue is very "cosmetic" one, but its existence
>>>> conflicts with my perfectionism so I'd like to resolve it ;)
>>>>
>>>>>
>>>>>>
>>>>>> Also, on one cluster with post-1.1.7 with openais plugin I have 16 nodes
>>>>>> configured in totem.interface.members, but only three nodes in <nodes>
>>>>>> CIB section, And I'm able to allocate at least 8-9 instances of clones
>>>>>> with clone-max.
>>>>>
>>>>> Yes, but did you set clone-node-max?  One is the global maximum, the
>>>>> other is the per-node maximum.
>>>>>
>>>>>> I believe that pacemaker does not query
>>>>>> totem.interface.members directly with openais plugin,
>>>>>
>>>>> Correct.
>>>>>
>>>>>> and
>>>>>> runtime.totem.pg.mrp.srp.members has only three nodes.
>>>>>> Did that behavior change recently?
>>>>>
>>>>> No.
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> For node removal we do require crm_node --remove.
>>>>>>>>>
>>>>>>>>> Is this not sufficient?
>>>>>>>>
>>>>>>>> I think it would be more straight-forward if there is only one origin of
>>>>>>>> membership information for entire cluster stack, so proposal is to
>>>>>>>> automatically remove node from CIB when it disappears from corosync
>>>>>>>> nodelist (due to removal by admin). That nodelist is not dynamic (read
>>>>>>>> from a config and then may be altered with cmapctl).
>>>>>>>
>>>>>>> Ok, but there still needs to be a trigger.
>>>>>>> Otherwise we waste cycles continuously polling corosync for something
>>>>>>> that is probably never going to happen.
>>>>>>
>>>>>> Please see above (cmap_track_add).
>>>>>>
>>>>>>>
>>>>>>> Btw. crm_node doesn't just remove the node from the cib, its existence
>>>>>>> is preserved in a number of caches which need to be purged.
>>>>>>
>>>>>> That could be done in a cmap_track_add's callback function too I think.
>>>>>>
>>>>>>> It could be possible to have crm_node also use the CMAP API to remove
>>>>>>> it from the running corosync, but something would still need to edit
>>>>>>> corosync.conf
>>>>>>
>>>>>> Yes, that is to admin.
>>>>>> Btw I think more about scenario Fabio explains in votequorum(8) in
>>>>>> 'allow_downscale' section - that is the one I'm interested in.
>>>>>>
>>>>>>>
>>>>>>> IIRC, pcs handles all three components (corosync.conf, CMAP, crm_node)
>>>>>>> as well as the "add" case.
>>>>>>
>>>>>> Good to know. But, I'm not ready yet to switch to it.
>>>>>>
>>>>>>>
>>>>>>>> Of course, it is possible to use crm_node to remove node from CIB too
>>>>>>>> after it disappeared from corosync, but that is not as elegant as
>>>>>>>> automatic one IMHO. And, that should not be very difficult to implement.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> utilizing
>>>>>>>>>> possibilities CMAP and votequorum provide.
>>>>>>>>>>
>>>>>>>>>> Idea is to:
>>>>>>>>>> * Do not add nodes from nodelist to CIB if their join-count in cmap is
>>>>>>>>>> zero (but do not touch CIB nodes which exist in a nodelist and have zero
>>>>>>>>>> join-count in cmap).
>>>>>>>>>> * Install watches on a cmap nodelist.node and
>>>>>>>>>> runtime.totem.pg.mrp.srp.members subtrees (cmap_track_add).
>>>>>>>>>> * Add missing nodes to CIB as soon as they are both
>>>>>>>>>> ** defined in a nodelist
>>>>>>>>>> ** their join count becomes non-zero.
>>>>>>>>>> * Remove nodes from CIB when they are removed from a nodelist.
>>>>>>>>>
>>>>>>>>> From _a_ nodelist or _the_ (optional) corosync nodelist?
>>>>>>>>
>>>>>>>> From the nodelist.node subtree of CMAP tree.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Because removing a node from the cluster because it shut down is... an
>>>>>>>>> interesting idea.
>>>>>>>>
>>>>>>>> BTW even that could be possible if quorum.allow_downscale is enabled,
>>>>>>>> but requires much more thinking and probably more functionality from
>>>>>>>> corosync. I'm not ready to comment on that yet though.
>>>>>>>
>>>>>>> "A node left but I still have quorum" is very different to "a node
>>>>>>> left... what node?".
>>>>>>> Also, what happens after you fence a node... do we forget about it too?
>>>>>>
>>>>>> quorum.allow_downscale mandates that it is active only if node leaves
>>>>>> the cluster in a clean state.
>>>>>>
>>>>>> But, from what I know, corosync does not remove node from a
>>>>>> nodelist.node neither itself nor on request from votequorum, that's why
>>>>>> I say about "more functionality from corosync".
>>>>>> If votequorum could distinguish "static" node (listed in config) from
>>>>>> "dynamic" node (added on-the-fly), and manage list of "dynamic" ones if
>>>>>> allow_downscale is enabled, that would do the trick.
>>>>>
>>>>> I doubt that request would get very far, but I could be wrong.
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I was about node removal from a CMAP's nodelist with corosync_cmapctl
>>>>>>>> command. Of course, absence of (optional) nodelist in CMAP would result
>>>>>>>> in NOOP because there is no removal event on a nodelist.node tree from cmap.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Certainly, this requires some CMAP values (especially votequorum ones
>>>>>>>>>> and may be totem mode) to have some 'well-known' values, f.e. only UDPU
>>>>>>>>>> mode and quorum.allow_downscale=1, that should be defined yet.
>>>>>>>>>>
>>>>>>>>>> May be, it also have sense to make this depend on some new CMAP
>>>>>>>>>> variable, f.e. nodelist.dynamic=1.
>>>>>>>>>>
>>>>>>>>>> I would even try to implement this if general agreement is gained and
>>>>>>>>>> nobody else wants to implement this.
>>>>>>>>>>
>>>>>>>>>> Can you please comment on this?
>>>>>>>>>>
>>>>>>>>>> Vladislav
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

More information about the Pacemaker mailing list