[Pacemaker] one node not always joining the 3-node cluster[Welcome reply not received from: node3]

[ESWAR RAO]

Hi,

I have a 3 node cluster [node1 ,node2, node3] HB+pacemaker setup.
All config files and auth files are same on all the 3 nodes.

But strangely always node3 is unable to join cluster group.
In the ha logs of DC node, node2 I could see below logs:

Jun 03 11:19:27 node2 crmd: [9637]: ERROR: crm_timer_popped: Integration
Timer (I_INTEGRATED) just popped in state S_INTEGRATION! (180000ms)
Jun 03 11:19:27 node2crmd: [9637]: info: crm_timer_popped: Welcomed: 1,
Integrated: 1
Jun 03 11:19:27 node2 crmd: [9637]: notice: do_state_transition:
Statetransition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 03 11:19:27 node2 crmd: [9637]: WARN: do_state_transition: Progressed
to state S_FINALIZE_JOIN after C_TIMER_POPPED
Jun 03 11:19:27 node2 crmd: [9637]: WARN: do_state_transition: 1 cluster
nodes failed to respond to the join offer.
Jun 03 11:19:27 node2 crmd: [9637]: info: ghash_print_node:   Welcome reply
not received from: node3 2


To recover, always I remove contents in /var/lib/heartbeat/crm and
/var/lib/heartbeat/hostcache and restart heartbeat on node3 then it joins
cluster group.

Thanks
Eswar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20140610/aaddf4d2/attachment.html>