[Pacemaker] Pacemaker 1.1.8, Corosync, No CMAN, Promotion issues

[Andrew Beekhof]

On 12/04/2013, at 12:11 PM, pavan tc <pavan.tc at gmail.com> wrote:

> Hi Andrew,
> 
> Thanks much for looking at this.
> 
> 
> > Then (after about 15 minutes), I see the following:
> 
> There were no logs at all in between?
> 
> Absolutely none in the syslog. Only the regular monitor logs from my resource agent which continued to report as secondary.

This is very strange, because the thing that caused the I_PE_CALC is a timer that goes off every 15 minutes.
Which would seem to imply that there was a transition of some kind about when the failure happened - but somehow it didnt go into the logs.

Could you post the complete logs from 14:00 to 14:30?

> I also checked /var/log/cluster/corosync.log. The only difference between this and the ones in syslog are the messages below:
> 
> From /var/log/cluster/corosync.log:
> -------------------------------------------------------
> Apr 10 14:12:38 [3391] vsanqa4       crmd:   notice: ais_dispatch_message:      Membership 166060: quorum lost
> Apr 10 14:12:38 [3386] vsanqa4        cib:   notice: crm_update_peer_state:     crm_update_ais_node: Node vsanqa3[1950617772] - state is now lost
> Apr 10 14:12:38 [3391] vsanqa4       crmd:   notice: crm_update_peer_state:     crm_update_ais_node: Node vsanqa3[1950617772] - state is now lost
> Apr 10 14:12:38 [3391] vsanqa4       crmd:     info: peer_update_callback:      vsanqa3 is now lost (was member)
> Apr 10 14:12:38 corosync [CPG   ] chosen downlist: sender r(0) ip(172.16.68.117) ; members(old:2 left:1)
> Apr 10 14:12:38 corosync [MAIN  ] Completed service synchronization, ready to provide service.
> 
> Apr 10 14:12:38 [3386] vsanqa4        cib:     info: cib_process_request:       Operation complete: op cib_modify for section nodes (origin=local/crmd/62, version=0.668.12): OK (rc=0)
> Apr 10 14:12:38 [3386] vsanqa4        cib:     info: cib_process_request:       Operation complete: op cib_modify for section cib (origin=local/crmd/64, version=0.668.14): OK (rc=0)
> Apr 10 14:12:38 [3391] vsanqa4       crmd:     info: crmd_ais_dispatch:         Setting expected votes to 2
> Apr 10 14:12:38 [3386] vsanqa4        cib:     info: cib_process_request:       Operation complete: op cib_modify for section crm_config (origin=local/crmd/66, version=0.668.15): OK (rc=0)
> 
> The first six out of the 10 messages above were seen on syslog too. Adding them here for context. The last four are the "extra" messages in corosync.log
> 
> Pavan
> 
> 
> >
> > Apr 10 14:26:46 vsanqa4 crmd[3391]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
> > Apr 10 14:26:46 vsanqa4 pengine[3390]:   notice: unpack_config: On loss of CCM Quorum: Ignore
> > Apr 10 14:26:46 vsanqa4 pengine[3390]:   notice: LogActions: Promote vha-6f92a1f6-969c-4c41-b9ca-7eb6f83ace2e:0#011(Slave -> Master vsanqa4)
> > Apr 10 14:26:46 vsanqa4 pengine[3390]:   notice: process_pe_message: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-input-392.bz2
> >
> > Thanks,
> > Pavan
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org