[Pacemaker] [Cluster-devel] [PATCH] Fix pacemaker's wrong quorum view in a CMAN+pacemaker cluster

Mon Mar 14 15:37:43 UTC 2011

On 03/14/2011 11:41 AM, Andrew Beekhof wrote:
> On Mon, Mar 14, 2011 at 11:04 AM, Simone Gotti <simone.gotti at gmail.com> wrote:
>> On 03/14/2011 08:36 AM, Andrew Beekhof wrote:
>>> On Sun, Mar 13, 2011 at 2:17 PM, Simone Gotti <simone.gotti at gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> Testing a cman+pacemaker cluster on rhel6 I noticed a very nasty
>>>> behavior when some nodes were leaving and rejoining the cluster. When a
>>>> nodes starts leaving and rejoining the cluster the quorum view of
>>>> pacemaker starts becoming sometimes different from the quorum view of
>>>> cman. The one not telling the truth was pacemaker.
>>> Do they ever start agreeing again?  In other words, is the situation
>>> transient or is Pacemaker always 1 (or more) behind after that?
>> Looks like it will be always behind and even more at every new cluster
>> change. I don't see any other part in the code where CMAN's events are
>> dequeued other than this one.
> Ok, it looks like the functions behind G_main_add_fd() don't work as
> one would expect.
> This is a very surprising thing to find out after all these years.
>
> Patch applied. Thanks!
> Could I trouble you to create a RHEL6 bug for this though?
> That will allow me to fix it there too.

Done. https://bugzilla.redhat.com/show_bug.cgi?id=684825

Thanks!

>> The event just says if we have or not quorum. So can happen that an old
>> dequeued message says the same as the current real state but it's just a
>> coincidence.
>>
>>
>>
>>>> I reproduced the problem with a simple test case made of 2 nodes using
>>>> cman (no two_nodes flag) and pacemaker (started only on the first node:
>>>> pcmk01).
>>>>
>>>> For the tests I was using the latest version of pacemaker (1.1.5) while
>>>> keeping the original versions of corosync and cluster (cman) packages
>>>> provided by the rhel6 (corosync-1.2.3-21.el6.x86_64,
>>>> cman-3.0.12-23.el6.4.x86_64)
>>>>
>>>> The problem is that when a node joins a cluster (starting cman) the cman
>>>> on the other nodes emits not one but 2 events (I didn't investigated if
>>>> this is normal or present only in some versions of cman) but when crmd
>>>> calls cman_dispatch it's using the flag CMAN_DISPATCH_ONE so only one of
>>>> the two events is dequeued. In the subsequent cluster event the old one
>>>> is dequeued.
>>>>
>>>> The fix I tried used CMAN_DISPATCH_ALL instead of CMAN_DISPATCH_ONE and
>>>> looks like its working.
>>>>
>>>> I'm CCing the cluster-devel list as they can be interested in the double
>>>> event emitted by cman.
>>>>
>>>>
>>>> Thanks.
>>>>
>>>> Bye!
>>>>
>>>>
>>>> == Test case ==
>>>>
>>>> === Without the patch ===
>>>>
>>>> Start with both nodes with cman started (so the cluster is quorate).
>>>>
>>>>
>>>> Now stop cman on pcmk02. Output on pcmk01:
>>>>
>>>> pcmk01 corosync[16793]:   [CMAN  ] quorum lost, blocking activity
>>>> pcmk01 corosync[16793]:   [QUORUM] This node is within the non-primary
>>>> component and will NOT provide any services.
>>>> pcmk01 corosync[16793]:   [QUORUM] Members[1]: 1
>>>> pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
>>>> membership and a new membership was formed.
>>>> pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 1
>>>> pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
>>>> ip(192.168.200.71)
>>>> pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
>>>> ready to provide service.
>>>> pcmk01 crmd: [16993]: notice: cman_event_callback: Membership 668:
>>>> quorum lost
>>>>
>>>> Only one event is enqueued.
>>>>
>>>> Now start again cman on pcmk02. Output on pcmk01:
>>>>
>>>> pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
>>>> membership and a new membership was formed.
>>>> pcmk01 corosync[16793]:   [CMAN  ] quorum regained, resuming activity
>>>> pcmk01 corosync[16793]:   [QUORUM] This node is within the primary
>>>> component and will provide service.
>>>> pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
>>>> pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
>>>> pcmk01 crmd: [16993]: notice: cman_event_callback: Membership 672:
>>>> quorum acquired
>>>> pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
>>>> pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
>>>> pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
>>>> ip(192.168.200.71)
>>>> pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
>>>> ready to provide service.
>>>>
>>>> As you can see two events are enqueued and only one si dequeued (due to
>>>> the CMAN_DISPATCH_ONE flag passed to cman_dispatch).
>>>>
>>>> The quorum is ragained both on cman and crmd. But there's another event
>>>> saying that the quorum is regained in the queue.
>>>>
>>>>
>>>> Now stop again cman on pcmk02. Output on pcmk01:
>>>>
>>>> pcmk01 corosync[16793]:   [CMAN  ] quorum lost, blocking activity
>>>> pcmk01 corosync[16793]:   [QUORUM] This node is within the non-primary
>>>> component and will NOT provide any services.
>>>> pcmk01 corosync[16793]:   [QUORUM] Members[1]: 1
>>>> pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
>>>> membership and a new membership was formed.
>>>> pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 1
>>>> pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
>>>> ip(192.168.200.71)
>>>> pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
>>>> ready to provide service.
>>>> pcmk01 crmd: [16993]: info: cman_event_callback: Membership 676: quorum
>>>> retained
>>>>
>>>> CMAN says that the quorum is lost and only one event is dispatched. But
>>>> crmd dequeued the previous event and thinks that we have the quorum.
>>>>
>>>>
>>>> Now start again cman on pcmk02. Output on pcmk01:
>>>>
>>>> pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
>>>> membership and a new membership was formed.
>>>> pcmk01 corosync[16793]:   [CMAN  ] quorum regained, resuming activity
>>>> pcmk01 corosync[16793]:   [QUORUM] This node is within the primary
>>>> component and will provide service.
>>>> pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
>>>> pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
>>>> pcmk01 crmd: [16993]: notice: cman_event_callback: Membership 680:
>>>> quorum lost
>>>> pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
>>>> pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
>>>> pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
>>>> ip(192.168.200.71)
>>>> pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
>>>> ready to provide service.
>>>>
>>>> CMAN says that the quorum is regained but crmd dequeued again the old
>>>> event and now it says that the quorum is lost. And so on...
>>>>
>>>>
>>>>
>>>> === With the patch ===
>>>>
>>>> stop cman on pcmk02. Output on pcmk01:
>>>>
>>>> pcmk01 corosync[13149]:   [CMAN  ] quorum lost, blocking activity
>>>> pcmk01 corosync[13149]:   [QUORUM] This node is within the non-primary
>>>> component and will NOT provide any services.
>>>> pcmk01 corosync[13149]:   [QUORUM] Members[1]: 1
>>>> pcmk01 corosync[13149]:   [TOTEM ] A processor joined or left the
>>>> membership and a new membership was formed.
>>>> pcmk01 corosync[13149]:   [CPG   ] downlist received left_list: 1
>>>> pcmk01 corosync[13149]:   [CPG   ] chosen downlist from node r(0)
>>>> ip(192.168.200.71)
>>>> pcmk01 corosync[13149]:   [MAIN  ] Completed service synchronization,
>>>> ready to provide service.
>>>>
>>>>  pcmk01 crmd: [13351]: notice: cman_event_callback: Membership 648:
>>>> quorum lost
>>>>
>>>> Only one event is enqued.
>>>>
>>>>
>>>> Now start again cman on pcmk02. Output on pcmk01:
>>>>
>>>> pcmk01 corosync[13149]:   [TOTEM ] A processor joined or left the
>>>> membership and a new membership was formed.
>>>> pcmk01 corosync[13149]:   [CMAN  ] quorum regained, resuming activity
>>>> pcmk01 corosync[13149]:   [QUORUM] This node is within the primary
>>>> component and will provide service.
>>>> pcmk01 corosync[13149]:   [QUORUM] Members[2]: 1 2
>>>> pcmk01 corosync[13149]:   [QUORUM] Members[2]: 1 2
>>>> pcmk01 crmd: [13351]: notice: cman_event_callback: Membership 652:
>>>> quorum acquired
>>>> pcmk01 corosync[13149]:   [CPG   ] downlist received left_list: 0
>>>> pcmk01 corosync[13149]:   [CPG   ] downlist received left_list: 0
>>>> pcmk01 corosync[13149]:   [CPG   ] chosen downlist from node r(0)
>>>> ip(192.168.200.71)
>>>> pcmk01 corosync[13149]:   [MAIN  ] Completed service synchronization,
>>>> ready to provide service.
>>>> pcmk01 crmd: [13351]: info: cman_event_callback: Membership 652: quorum
>>>> retained
>>>>
>>>> As you can see two events are enqued and both are dequeued.
>>>>
>>>>
>>>>
>>>> --
>>>> Simone Gotti
>>>>
>>>>
>>>>
>>
>> --
>> Simone Gotti
>>
>>