[Pacemaker] Remove a "ghost" node

[Andrew Beekhof]

On 8 Nov 2013, at 4:45 am, Sean Lutner <sean at rentul.net> wrote:

> I have a confusing situation that I'm hoping to get help with. Last night after configuring STONITH on my two node cluster, I suddenly have a "ghost" node in my cluster. I'm looking to understand the best way to remove this node from the config.
> 
> I'm using the fence_ec2 device for for STONITH. I dropped the script on each node, registered the device with stonith_admin -R -a fence_ec2 and confirmed the registration with both
> 
> # stonith_admin -I
> # pcs stonith list
> 
> I then configured STONITH per the Clusters from Scratch doc
> 
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_example.html
> 
> Here are my commands:
> # pcs cluster cib stonith_cfg
> # pcs -f stonith_cfg stonith create ec2-fencing fence_ec2 ec2-home="/opt/ec2-api-tools" pcmk_host_check="static-list" pcmk_host_list="ip-10-50-3-122 ip-10-50-3-251" op monitor interval="300s" timeout="150s" op start start-delay="30s" interval="0"
> # pcs -f stonith_cfg stonith
> # pcs -f stonith_cfg property set stonith-enabled=true
> # pcs -f stonith_cfg property
> # pcs cluster push cib stonith_cfg
> 
> After that I saw that STONITH appears to be functioning but a new node listed in pcs status output:

Do the EC2 instances have fixed IPs?
I didn't have much luck with EC2 because every time they came back up it was with a new name/address which confused corosync and created situations like this.

> 
> # pcs status
> Last updated: Thu Nov  7 17:41:21 2013
> Last change: Thu Nov  7 04:29:06 2013 via cibadmin on ip-10-50-3-122
> Stack: cman
> Current DC: ip-10-50-3-122 - partition with quorum
> Version: 1.1.8-7.el6-394e906
> 3 Nodes configured, unknown expected votes
> 11 Resources configured.
> 
> 
> Node ip-10-50-3-1251: UNCLEAN (offline)
> Online: [ ip-10-50-3-122 ip-10-50-3-251 ]
> 
> Full list of resources:
> 
> ClusterEIP_54.215.143.166      (ocf::pacemaker:EIP):   Started ip-10-50-3-122
> Clone Set: EIP-AND-VARNISH-clone [EIP-AND-VARNISH]
>     Started: [ ip-10-50-3-122 ip-10-50-3-251 ]
>     Stopped: [ EIP-AND-VARNISH:2 ]
> ec2-fencing    (stonith:fence_ec2):    Stopped 
> 
> I have no idea where the node that is marked UNCLEAN came from, though it's a clear typo is a proper cluster node.
> 
> The only command I ran with the bad node ID was:
> 
> # crm_resource --resource ClusterEIP_54.215.143.166 --cleanup --node ip-10-50-3-1251
> 
> Is there any possible way that could have caused the the node to be added?
> 
> I tried running pcs cluster node remove ip-10-50-3-1251 but since there is no node and thus no pcsd that failed. Is there a way I can safely remove this ghost node from the cluster? I can provide logs from pacemaker or corosync as needed.
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org