[Pacemaker] corosync vs. pacemaker 1.1

[Andrew Beekhof]

On Sat, Feb 11, 2012 at 2:46 AM, Kiss Bence <bence at noc.elte.hu> wrote:
> Hi,
>
>
> On 01/30/2012 04:00 AM, Andrew Beekhof wrote:
>>
>> On Thu, Jan 26, 2012 at 2:08 AM, Kiss Bence<bence at noc.elte.hu>  wrote:
>>>
>>> Hi,
>>>
>>> I am newbie to the clustering and I am trying to build a two node
>>> active/passive cluster based upon the documentation:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>
>>> My systems are Fedora 14, uptodate. After forming the cluster as wrote, I
>>> started to test it. (resources: drbd->  lvm->  fs ->group of services)
>>> Resources moved around, nodes rebooted and killed (first I tried it in
>>> virtual environment then also on real machines).
>>>
>>> After some events the two nodes ended up in a kind of state of
>>> split-brain.
>>> The crm_mon showed me that the other node is offline at both nodes
>>> although
>>> the drbd subsystem showed everything in sync and working. The network was
>>> not the issue (ping, tcp and udp communications were fine). Nothing
>>> changed
>>> from the network view.
>>>
>>> At first the rejoining took place quiet well, but some more events after
>>> it
>>> took longer and after more event it didn't. The network dump showed me
>>> the
>>> multicast packets still coming and going. At corosync (crm_node -l) the
>>> other node didn't appeared both on them. After trying configuring the cib
>>> logs was full of messages like "<the other node>: not in our membership".
>>
>>
>> That looks like a pacemaker bug.
>> Can you use crm_report to grab logs from about 30 minutes prior to the
>> first time you see this log until an hour after please?
>>
>> Attach that to a bug in bugs.clusterlabs.org and i'll take a look
>
>
> I had created a bug report: id 5031.

Perfect, I'll look there.

>
> The "split-brain" is lasting every time about 5 minutes. Meanwhile the two
> nodes think that the other node is dead. However the drbd is working fine,
> and properly disallowing the second rebooted node to go Primary. The
> crm_node -l shows only the local node.
>
> Meanwhile one of my question is answered. The multicast issue was a local
> network issue. The local netadmin fixed it. Now it works.
>
> This issue seems to me similar to what James Flatten had reported at 8-th
> Feb. ([Pacemaker] Question about cluster start-up in a 2 node cluster with a
> node offline.)
>
> The stonith-enabled="false" \
> and no-quorum-policy="ignore"
>
> Thanks in advance,
> Bence
>
>
>
>>
>>>
>>> I tried to erase the config (crm configure erase, cibadmin -E -f) but it
>>> worked only locally. I noticed that the pacemaker process didn't started
>>> up
>>> normally on the node that was booting after the other. I also tried to
>>> remove files from /var/lib/pengine/ and /var/lib/hearbeat/crm/ but only
>>> the
>>> resources are gone. It didn't help on forming a cluster without
>>> resources.
>>> The pacemaker process exited some 20 minutes after it started. Manual
>>> starting was the same.
>>>
>>> After digging into google for answers I found nothing helpful. From
>>> running
>>> tips I changed in the /etc/corosync/service.d/pcmk file the version to
>>> 1.1
>>> (this is the version of the pacemaker in this distro). I realized that
>>> the
>>> cluster processes were startup from corosync itself not by pacemaker.
>>> Which
>>> could be omitted. The cluster forming is stable after this change even
>>> after
>>> many many events.
>>>
>>> Now I reread the document mentioned above, and I wonder why it wrote the
>>> "Important notice" on page 37. What is wrong theoretically with my
>>> scenario?
>>
>>
>> Having corosync start the daemons worked well for some but not others,
>> thus it was unreliable.
>> The notice points out a major difference between the two operating
>> modes so that people will not be caught by surprise when pacemaker
>> does not start.
>>
>>> Why does it working? Why didn't work the config suggested by the
>>> document?
>>>
>>> Tests were done firsth on virtual machines of a Fedora 14 (1 CPU core,
>>> 512Mb
>>> ram, 10G disk, 1G drbd on logical volume, physical  volume on drbd
>>> forming
>>> volgroup named cluster.)/node.
>>>
>>> Then on real machines. They have more cpu cores (4), more RAM (4G) and
>>> more
>>> disk (mirrored 750G), 180G drbd, and 100M garanteed routed link between
>>> the
>>> nodes 5 hops away.
>>>
>>> By the way how should one configure the corosync to work on multicast
>>> routed
>>> network? I had to create an openvpn tap link between the real nodes for
>>> working. The original config with public IP-s didn't worked. Is corosync
>>> equipped to cope with the multicast pim messages? Or it was a firewall
>>> issue.
>>>
>>> Thanks in advance,
>>> Bence
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org