[Pacemaker] Removed nodes showing back in status

[David Vossel]

----- Original Message -----
> From: "Larry Brigman" <larry.brigman at gmail.com>
> To: "The Pacemaker cluster resource manager" <pacemaker at oss.clusterlabs.org>
> Sent: Monday, May 14, 2012 1:30:22 PM
> Subject: Re: [Pacemaker] Removed nodes showing back in status
> 
> On Mon, May 14, 2012 at 9:54 AM, Larry Brigman
> <larry.brigman at gmail.com> wrote:
> > I have a 5 node cluster (but it could be any number of nodes, 3 or
> > larger).
> > I am testing some scripts for node removal.
> > I remove a node from the cluster and everything looks correct from
> > crm
> > status standpoint.
> > When I remove a second node, the first node that was removed now
> > shows back
> > in the crm status as off-line.  I'm following the guidelines
> > provided
> > in Pacemaker Explained docs.
> > http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-delete.html
> >
> > I believe this is a bug but want to put it out to the list to be
> > sure.
> > Versions.
> > RHEL5.7 x86_64
> > corosync-1.4.2
> > openais-1.1.3
> > pacemaker-1.1.5
> >
> > Status after first node removed
> > [root at portland-3 ~]# crm status
> > ============
> > Last updated: Mon May 14 08:42:04 2012
> > Stack: openais
> > Current DC: portland-1 - partition with quorum
> > Version: 1.1.5-1.3.sme-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
> > 4 Nodes configured, 4 expected votes
> > 0 Resources configured.
> > ============
> >
> > Online: [ portland-1 portland-2 portland-3 portland-4 ]
> >
> > Status after second node removed.
> > [root at portland-3 ~]# crm status
> > ============
> > Last updated: Mon May 14 08:42:45 2012
> > Stack: openais
> > Current DC: portland-1 - partition with quorum
> > Version: 1.1.5-1.3.sme-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
> > 4 Nodes configured, 3 expected votes
> > 0 Resources configured.
> > ============
> >
> > Online: [ portland-1 portland-3 portland-4 ]
> > OFFLINE: [ portland-5 ]
> >
> > Both nodes were removed from the cluster from node 1.
> 
> When I added a node back into the cluster the second node
> that was removed now shows as offline.

The only time I've seen this sort of behavior is when I don't completely shutdown corosync and pacemaker on the node I'm removing before I delete it's configuration from the cib.  Are you sure corosync and pacemaker are gone before you delete the node from the cluster config?

-- Vossel

> [root at portland-3 ~]# crm status
> ============
> Last updated: Mon May 14 11:27:55 2012
> Stack: openais
> Current DC: portland-1 - partition with quorum
> Version: 1.1.5-1.3.sme-01e86afaaa6d4a8c4836f68df80ababd6ca3902f
> 5 Nodes configured, 4 expected votes
> 0 Resources configured.
> ============
> 
> Online: [ portland-1 portland-3 portland-4 portland-5 ]
> OFFLINE: [ portland-2 ]
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>