[Pacemaker] Cluster split brain on vmware VSphere

[Torresani, Roberto]

Hi list,
 sorry if this is not the right ml or if the question is already answered somewhere...if this is the case, just point me to the solution please.

I have two active/passive mysql cluster on vmware vsphere running:
- CentOS 5.5 32 bit
- heartbeat-3.0.3-2.el5
- corosync-1.2.2-1.1.el5
- pacemaker-1.0.8-6.1.el5

The cluster have stonith configured using an adapted version of xen0 stonith external plugins, which seems to work fine.

The problem is that from time to time (every day...), especially when the virtualcenter is snapshotting the vm for make backup of them, the cluster split itself.
Here some of the events:

Jun  5 22:06:12 mysqldrupal-n1 corosync[3139]:   [TOTEM ] Process pause detected for 1111 ms, flushing membership messages.
Jun  5 22:06:12 mysqldrupal-n1 corosync[3139]:   [TOTEM ] Process pause detected for 1112 ms, flushing membership messages.
Jun  5 22:06:12 mysqldrupal-n1 corosync[3139]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jun  5 22:16:18 mysqldrupal-n1 crmd: [3151]: ERROR: crmd_ha_msg_filter: Another DC detected: mysqldrupal-n2 (op=noop)

and both nodes start resetting one each other, till the situation stabilizes after a while.

I tried to modify some parameters in the corosync.conf but without luck...

Both cluster behaves the same.

Anyone of you have similar problem?
Any idea please?

In attach you can find the relevant messages log and the cib configuration.

Regards,
Roberto
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: messages.txt
URL: <http://lists.clusterlabs.org/pipermail/pacemaker/attachments/20100607/585390f2/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: cib.txt
URL: <http://lists.clusterlabs.org/pipermail/pacemaker/attachments/20100607/585390f2/attachment-0005.txt>