[Pacemaker] Periodically appear non-existent nodes

[David Vossel]

----- Original Message -----
> From: "ruslan usifov" <ruslan.usifov at gmail.com>
> To: "The Pacemaker cluster resource manager" <pacemaker at oss.clusterlabs.org>
> Sent: Tuesday, April 17, 2012 6:46:00 AM
> Subject: Re: [Pacemaker] Periodically appear non-existent nodes
> 
> 
> 2012/4/17 Andreas Kurz < andreas at hastexo.com >
> 
> 
> 
> 
> On 04/14/2012 11:14 PM, ruslan usifov wrote:
> > Hello
> > 
> > I remove 2 nodes from cluster, with follow sequence:
> > 
> > crm_node --force -R <id of node1>
> > crm_node --force -R <id of node2>
> > cibadmin --delete --obj_type nodes --crm_xml '<node
> > uname="node1"/>'
> > cibadmin --delete --obj_type status --crm_xml '<node_state
> > uname="node1"/>'
> > cibadmin --delete --obj_type nodes --crm_xml '<node
> > uname="node2"/>'
> > cibadmin --delete --obj_type status --crm_xml '<node_state
> > uname="node2"/>'
> > 
> > 
> > Nodes after this deleted, but if for example i restart (reboot) one
> > of
> > existent nodes in working cluster, this deleted nodes appear again
> > in
> > OFFLINE state
> 
> Just to double check ... corosync was already stopped (on these
> to-be-deleted nodes) prior to the deletion and it's still stopped on
> the
> removed nodes? ... and no cman involved?
> 
> 
> This nodes doesn't present physically:-)) (we remove this from
> network), so no corosync no cman not anything else

I don't know if this is what you are experiencing, but here is one explanation that I can easily reproduce.

If you remove the node from the CIB then disconnect the node from the network while corosync is running on the node,  a loss of membership will be detected by corosync on the remaining nodes.  Pacemaker on the other nodes will get a message from corosync saying node membership changed with the id of the node that left the cluster.  Pacemaker then says, hey we know about this node that isn't online which will re-populate some of the fields in the CIB you just deleted.

-- Vossel