[Pacemaker] node can't join cluster after reboot

[Andrew Beekhof]

On Tue, Oct 30, 2012 at 4:35 PM, Vladimir Elisseev <vovan at vovan.nl> wrote:
> Thanks for trying to help! Currently I can't provide crm_report from the
> failed node, as I've decided to restore the complete node from backup.
> The versions I use are corosync-1.3.0 and pacemaker-1.0.10. Actually the
> problem occurred after updating quiet a few system packages, but all the
> cluster related software was untouched. I've found exactly the same
> issue described in the mailing list earlier:
> http://www.gossamer-threads.com/lists/linuxha/pacemaker/77881?do=post_view_threaded#77881
> At least symptoms are exactly the same as well as pasted log files. I've
> tried enable debug logging as well and saw that crm tries to connect to
> cib sockets (/var/run/crm_*) too early (IMO) and fails because cib
> wasn't started yet.
> I'm planning to repeat update of these system again, but I'll do this
> more carefully in order to understand which particular package leads to
> this behavior. BTW, how can I create crm_report? I can't find this
> binary anywhere on the system.

Its included in subsequent 1.0.x releases.
You should have hb_report available though.

> Let me know what kind of input you'll
> need if I'll be able to reproduce this problem.
>
> Regards,
> Vlad.
>
>
> On Tue, 2012-10-30 at 16:00 +1100, Andrew Beekhof wrote:
>> On Sun, Oct 28, 2012 at 9:05 PM, Vladimir Elisseev <vovan at vovan.nl> wrote:
>> > Hello,
>> >
>> > I'm having problem that after reboot one cluster node can't join cluster
>> > anymore. Form the log file I can't understand what actually is going on.
>> > I only can see, that cib and crm both are respawned frequently. I'd
>> > appreciate any help. Below is relevant part of the log file:
>>
>> I appreciate that you're trying to keep it brief, but problems often
>> originate much earlier than people suspect.
>> Can you instead attach a crm_report tarball, that will have everything
>> (from both nodes) that we need to be able to help.
>>
>> What version is this btw?
>>
>> >
>> > Oct 28 10:52:22 srv2 cib: [10646]: info: cib_server_process_diff: Requesting re-sync from peer
>> > Oct 28 10:52:22 srv2 cib: [10646]: WARN: cib_diff_notify: Local-only Change (client:crmd, call: 4770): -1.-1.-1 (Application of an update diff failed, requesting a full refresh)
>> > Oct 28 10:52:22 srv2 cib: [10653]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.qJTUAV (digest: /var/lib/heartbeat/crm/cib.XwOKXQ)
>> > Oct 28 10:52:22 srv2 cib: [10646]: WARN: cib_server_process_diff: Not applying diff 0.1298.5 -> 0.1299.1 (sync in progress)
>> > Oct 28 10:52:22 srv2 cib: [10646]: info: cib_replace_notify: Local-only Replace: -1.-1.-1 from srv1
>> > Oct 28 10:52:22 corosync [pcmk]:  ] info: pcmk_ipc_exit: Client cib (conn=0x1837340, async-conn=0x1837340) left
>> > Oct 28 10:52:22 corosync [pcmk]:  ] ERROR: pcmk_wait_dispatch: Child process cib terminated with signal 6 (pid=10646, core=true)
>> > Oct 28 10:52:22 corosync [pcmk]:  ] notice: pcmk_wait_dispatch: Respawning failed child process: cib
>> > Oct 28 10:52:22 corosync [pcmk]:  ] info: spawn_child: Forked child 10656 for process cib
>> > Oct 28 10:52:22 srv2 cib: [10656]: info: Invoked: /usr/lib64/heartbeat/cib
>> >
>> >
>> > Regards,
>> > Vlad.
>> >
>> >
>> > _______________________________________________
>> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >
>> > Project Home: http://www.clusterlabs.org
>> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> > Bugs: http://bugs.clusterlabs.org
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org