[Pacemaker] Node recover causes resource to migrate

[Andrew Beekhof]

On 23/07/2013, at 2:20 AM, Jacobo García <jacobo.garcia at gmail.com> wrote:

> Hello,
> 
> I'm trying the debug the following scenario:
> 
> My cluster setup is two AWS instances providing an elastic ip as an HA resource. I have written a custom resource script for managing that, the script passes all the tests specified in ocf-tester. It behaves properly in other test scenarios. 
> 
> In this link[1] you can find the corosync configuration file of node A and the pacemaker configuration as well. The corosync configuration of the node B is reciprocal.
> 
> But I founding lots of behavior that I am not able to understand in the following situation:
> 
> 1. Both nodes are up and node A has the Elastic IP
> 2. I turn off node A through AWS API.
> 3. After some time (around 2-3 mins) the Elastic IP moves to node B.
> 
> The strange behavior starts when I try to bring the node A back to the cluster:
> 
> 4. Node A is back with a different internal ip address.

This is your basic problem.

I don't believe there is any cluster software that is designed to support this sort of scenario.
Even at the corosync level, it has no knowledge that this is the same machine that left. 

Take a look at Amazon's virtual private network feature.
That lets you assign static IPs to the instances that make up your cluster.

> 5. I update the ip address in the A node: `crm configure edit` 
> 6. I add the ip address of node A in  node B: `crm configure edit` (deleting the old ip of node B is not possible).
> 7. I update the ip address in the corosync configuration of node A.
> 8. I restart corosync in node A.
> 9. IP migrates from node B to node A because node B appears offline to node A, this is because node A has not had its corosync config updated because I don't want to restart corosync on the resource that is providing the ip.
> 10. At this point node A only sees itself and node B only sees itself.
> 
> Here is the log files of both nodes[2].
> 
> How can I bring a node back without having it moving the IP address back? 
> 
> Thanks in advance.
> 
> [1] https://gist.github.com/therobot/67bae18b5acb0ba78925
> [2] https://gist.github.com/therobot/ef85646d6e0092e494c4
> 
> 
> 
> Jacobo García López de Araujo
> http://thebourbaki.com | http://twitter.com/clapkent 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org