[Pacemaker] custom resource-agent

[David Lang]

On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:

> On Tue, Oct 01, 2013 at 10:07:12AM -0700, David Lang wrote:
>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>
>>> On Tue, Oct 01, 2013 at 07:22:20AM -0700, David Lang wrote:
>>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>>
>>>>> Hi David,
>>>>>
>>>>> On Mon, Sep 30, 2013 at 12:41:23PM -0700, David Lang wrote:
>>>>>> On Mon, 30 Sep 2013, David Lang wrote:
>>>>>>
>>>>>>> On Mon, 30 Sep 2013, Michael Schwartzkopff wrote:
>>>>>>>
>>>>>>>> Am Montag, 30. September 2013, 21:12:56 schrieb Peter Romfeld:
>>>>>>>>> I am working in AWS i cant just use a VIP i need to use a floating
>>>>>>>>> secondary IP which i reassign through script, i want to let pacemaker
>>>>>>>>> handle the reassignment...
>>>>>>>>
>>>>>>>> Please explain the difference of a VIP and a "secondary IP" in
>>>>>>>> your opinion.
>>>>>>>
>>>>>>> with AWS you need to inform amazon of the change, not just change
>>>>>>> the IP on the local box, that requires much more work than a
>>>>>>> simple local VIP
>>>>>>
>>>>>> being more detailed, instead of just
>>>>>> ifconfig eth0:0 $vip
>>>>>> you have to do something like
>>>>>>
>>>>>> /opt/aws/bin/ec2-assign-private-ip-addresses -n $ENI_ID --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
>>>>>
>>>>> We may consider adding such an option to IPaddr2. Has anybody
>>>>> ever tried that?
>>>>>
>>>>>> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
>>>>>> if [ "$pingresult" == "0" ]; then
>>>>>> echo `date` "-- Restarting network"
>>>>>> /sbin/service network restart > /dev/null 2>&1
>>>>>
>>>>> That may break the cluster communication, which may lead to split
>>>>> brain, etc. Is that really the only way?
>>>>
>>>> It's not the only way, but you do have the problem that the call to
>>>> aws management interface is asynchronous, you don't know when it's
>>>> going to complete, and until it does, the IP doesn't actually work.
>>>
>>> Wouldn't it be then safer to wait until it starts working, i.e.
>>> to monitor in a loop?
>>
>> that's exactly what the snippet of code above is for, to detect when
>> the other box no longer has the address.
>
> Hmm, perhaps I'm missing something, but I couldn't notice a loop
> in that code. What I meant was something like this:
>
> while ! ping -c 1 -W 1 $VIP | grep -qs time=; do
> 	:
> done
>
> Then network restart wouldn't be necessary, right? Sorry, I don't
> know much about aws.

I haven't used this exact script before, but I have seen the problem that this 
script is designed to address. I am not saying that I agree with this script, 
but it's what Amazon is suggesting, so it's probably a reasonable start.


this was a cut-n-paste from the URL provided earlier 
http://aws.amazon.com/articles/2127188135977316


#!/bin/sh
# This script will monitor another HA node and take over a Virtual IP (VIP)
# if communication with the other node fails

# High Availability IP variables
# Other node's IP to ping and VIP to swap if other node goes down
HA_Node_IP=10.0.0.11
VIP=10.0.0.10

# Specify the EC2 region that this will be running in
REGION=us-west-2

# Run aws-apitools-common.sh to set up default environment variables and to
# leverage AWS security credentials provided by EC2 roles
. /etc/profile.d/aws-apitools-common.sh

# Determine the instance and ENI IDs so we can reassign the VIP to the
# correct ENI. Requires EC2 describe-instances and assign-private-ip-address
# permissions. The following example EC2 roles policy will authorize these
# commands:
# {
# "Statement": [
# {
# "Action": [
# "ec2:AssignPrivateIpAddresses",
# "ec2:DescribeInstances"
# ],
# "Effect": "Allow",
# "Resource": "*"
# }
# ]
# }

Instance_ID=`/usr/bin/curl --silent 
http://169.254.169.254/latest/meta-data/instance-id`ENI_ID=`/opt/aws/bin/ec2-describe-instances $Instance_ID --region $REGION | grep 
eni -m 1 | awk '{print $2;}'`

echo `date` "-- Starting HA monitor"
while [ . ]; do
  pingresult=`ping -c 3 -W 1 $HA_Node_IP | grep time= | wc -l`
  if [ "$pingresult" == "0" ]; then
  echo `date` "-- HA heartbeat failed, taking over VIP"
  /opt/aws/bin/ec2-assign-private-ip-addresses -n $ENI_ID --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
  pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
  if [ "$pingresult" == "0" ]; then
  echo `date` "-- Restarting network"
  /sbin/service network restart > /dev/null 2>&1
  fi
  sleep 60
  fi
  sleep 2
done


David Lang