[Pacemaker] custom resource-agent

David Lang david at lang.hm
Wed Oct 2 06:34:04 UTC 2013 

Unless something has changed in the AWS API in the last few months, when the aws 
command exits successfully, that doesn't mean the change has taken effect, just 
that the aws system has accepted the change and it will take effect 'soon'

David Lang

On Wed, 2 Oct 2013, Peter Romfeld wrote:

> yes i need to use the aws command, i am using a VPC, after issueing the
> command i get a "true" statement and its done
>
> so i only want pacemaker to issue the one-shot command at failover. Here is
> what i have atm: (i know its still dirty, just lerning pacemaker)
>
> primitive drbd_mysql ocf:linbit:drbd \
>        params drbd_resource="mydata" \
>        op monitor interval="15s"
> primitive fs_mysql ocf:heartbeat:Filesystem \
>        params device="/dev/drbdx" directory="/mountpint" fstype="ext4"
> options="relatime,barrier=1" \
>        op start interval="0" timeout="60" \
>        op stop interval="0" timeout="60" \
>        op monitor interval="10s" timeout="60s" OCF_CHECK_LEVEL="20" \
>        meta target-role="started"
> primitive fvip ocf:heartbeat:AWSFIP \
>        params fvip="192.168.2.10" region="ap-southeast-1"
> primitive ip_mysql ocf:heartbeat:IPaddr2 \
>        params ip="192.168.2.10" cidr_netmask="20" \
>        op monitor interval="10" \
>        meta target-role="started"
> primitive mysqld lsb:mysql
> group mysql fs_mysql ip_mysql mysqld
> ms ms_drbd_mysql drbd_mysql \
>        meta master-max="1" master-node-max="1" clone-max="2"
> clone-node-max="1" notify="true"
> colocation mysql_on_drbd inf: fvip mysql ms_drbd_mysql:Master
> order mysql_after_drbd_and_fvip inf: ms_drbd_mysql:promote fvip:start
> mysql:start
>
> my AWSFIP(adjusted Dummy :%s/dummy/awsfip/g|%s/Dummy/AWSFIP/g):
>
> <parameter name="fvip" unique="1" required="1">
> <longdesc lang="en">
> The IPv4 address to be configured in dotted quad notation, for example
> "192.168.1.1".
> </longdesc>
> <shortdesc lang="en">IPv4 address</shortdesc>
> <content type="string" default="" />
> </parameter>
>
> <parameter name="region" unique="1" required="1">
> <longdesc lang="en">
> The name of the AWS region
> </longdesc>
> <shortdesc lang="en">AWS region</shortdesc>
> <content type="string"/>
> </parameter>
>
> awsfip_start() {
>    awsfip_monitor
>    Instance_ID=`/usr/bin/curl --silent
> http://169.254.169.254/latest/meta-data/instance-id`
>    ENI_ID=`aws ec2 describe-instances --instance-id $Instance_ID --region
> $OCF_RESKEY_region | grep NetworkInterfaceId | cut -d '"' -f 4`
>    if [ $? =  $OCF_SUCCESS ]; then
>        return $OCF_SUCCESS
>    fi
>         aws ec2 assign-private-ip-addresses --network-interface-id $ENI_ID
> --private-ip-addresses $OCF_RESKEY_fvip --allow-reassignment --region
> $OCF_RESKEY_region
>         sleep 4
>         aws ec2 assign-private-ip-addresses --network-interface-id $ENI_ID
> --private-ip-addresses $OCF_RESKEY_fvip --allow-reassignment --region
> $OCF_RESKEY_region
>         /etc/init.d/networking restart
>    touch ${OCF_RESKEY_state}
> }
>
> I couldn't get it to work yet, and i don't want to run a external script
> for it. I can't be so hard to let pacemaker execute an additional one-shot
> command at failover (in the correct order..)
>
> Thanks for your help!
>
>
> On Wednesday, October 02, 2013 07:33 AM, David Lang wrote:
>
> the aws command is making the call to inform aws, if you were to bring up
> the address without making the aws command, would it work? If you are on a
> Virtual Private Cloud (VPC), it may, but I didn't think it would.
>
> If you can make it work without the aws command, then you can just use the
> standard pacemaker VIP configuration. I know that this doesn't work if you
> have an external IP that you are moving (you must use an aws call to tell
> Amazon to move the IP), but it's possible that you don't have to for an
> internal IP, but I would be surprised.
>
> David Lang
>
>
> On Wed, 2 Oct 2013, Peter Romfeld wrote:
>
> Hey,
> when i change the secondary IP per hand or with external script on a Ubuntu
> Instance I just need:
> /etc/network/interfaces
> auto eth0
> iface eth0 inet dhcp
>   address 192.168.32.12
>   netmask 255.255.240.0
>   gateway 192.168.32.1
>   up ip addr add 192.168.32.11/20 dev eth0
>
> and then run the script which basically just does:
> #!/bin/sh
>
> VIP=172.32.32.11
> REGION=ap-southeast-1
>
> Instance_ID=`/usr/bin/curl --silent
> http://169.254.169.254/latest/meta-data/instance-id`
> ENI_ID=`aws ec2 describe-instances --instance-id $Instance_ID --region
> $REGION | grep NetworkInterfaceId | cut -d '"' -f 4`
>
> aws ec2 assign-private-ip-addresses --network-interface-id $ENI_ID
> --private-ip-addresses $VIP --allow-reassignment --region $REGION
>
>
> I dont need to inform AWS or restart network, only the correct network
> config and the one command, when i tested it with pinging from a 3rd
> instance during IP change i didnt got any interupts. I dont know about
> monitoring it
>
>
> On Wed, Oct 2, 2013 at 1:38 AM, David Lang <david at lang.hm>
> <david at lang.hm>wrote:
>
> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>
> On Tue, Oct 01, 2013 at 10:07:12AM -0700, David Lang wrote:
>
>
> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>
> On Tue, Oct 01, 2013 at 07:22:20AM -0700, David Lang wrote:
>
>
> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>
> Hi David,
>
>
> On Mon, Sep 30, 2013 at 12:41:23PM -0700, David Lang wrote:
>
> On Mon, 30 Sep 2013, David Lang wrote:
>
> On Mon, 30 Sep 2013, Michael Schwartzkopff wrote:
>
>
> Am Montag, 30. September 2013, 21:12:56 schrieb Peter Romfeld:
>
>
> I am working in AWS i cant just use a VIP i need to use a floating
> secondary IP which i reassign through script, i want to let
> pacemaker
> handle the reassignment...
>
>
> Please explain the difference of a VIP and a "secondary IP" in
> your opinion.
>
>
> with AWS you need to inform amazon of the change, not just change
> the IP on the local box, that requires much more work than a
> simple local VIP
>
>
> being more detailed, instead of just
> ifconfig eth0:0 $vip
> you have to do something like
>
> /opt/aws/bin/ec2-assign-**private-ip-addresses -n $ENI_ID
> --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
>
>
> We may consider adding such an option to IPaddr2. Has anybody
> ever tried that?
>
> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
>
> if [ "$pingresult" == "0" ]; then
> echo `date` "-- Restarting network"
> /sbin/service network restart > /dev/null 2>&1
>
>
> That may break the cluster communication, which may lead to split
> brain, etc. Is that really the only way?
>
>
> It's not the only way, but you do have the problem that the call to
> aws management interface is asynchronous, you don't know when it's
> going to complete, and until it does, the IP doesn't actually work.
>
>
> Wouldn't it be then safer to wait until it starts working, i.e.
> to monitor in a loop?
>
>
> that's exactly what the snippet of code above is for, to detect when
> the other box no longer has the address.
>
>
> Hmm, perhaps I'm missing something, but I couldn't notice a loop
> in that code. What I meant was something like this:
>
> while ! ping -c 1 -W 1 $VIP | grep -qs time=; do
>        :
> done
>
> Then network restart wouldn't be necessary, right? Sorry, I don't
> know much about aws.
>
>
> I haven't used this exact script before, but I have seen the problem that
> this script is designed to address. I am not saying that I agree with this
> script, but it's what Amazon is suggesting, so it's probably a reasonable
> start.
>
>
> this was a cut-n-paste from the URL provided earlier
> http://aws.amazon.com/**articles/2127188135977316
> <http://aws.amazon.com/articles/2127188135977316><http://aws.amazon.com/articles/2127188135977316>
>
>
> #!/bin/sh
> # This script will monitor another HA node and take over a Virtual IP (VIP)
> # if communication with the other node fails
>
> # High Availability IP variables
> # Other node's IP to ping and VIP to swap if other node goes down
> HA_Node_IP=10.0.0.11
> VIP=10.0.0.10
>
> # Specify the EC2 region that this will be running in
> REGION=us-west-2
>
> # Run aws-apitools-common.sh to set up default environment variables and to
> # leverage AWS security credentials provided by EC2 roles
> . /etc/profile.d/aws-apitools-**common.sh
>
> # Determine the instance and ENI IDs so we can reassign the VIP to the
> # correct ENI. Requires EC2 describe-instances and
> assign-private-ip-address
> # permissions. The following example EC2 roles policy will authorize these
> # commands:
> # {
> # "Statement": [
> # {
> # "Action": [
> # "ec2:AssignPrivateIpAddresses"**,
> # "ec2:DescribeInstances"
> # ],
> # "Effect": "Allow",
> # "Resource": "*"
> # }
> # ]
> # }
>
> Instance_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/**
> meta-data/instance-id`ENI_ID=`**/opt/aws/bin/ec2-describe-**instances
> <http://169.254.169.254/latest/meta-data/instance-idENI_ID=/opt/aws/bin/ec2-describe-instances><http://169.254.169.254/latest/meta-data/instance-idENI_ID=/opt/aws/bin/ec2-describe-instances>$Instance_ID
> --region $REGION | grep eni -m 1 | awk '{print $2;}'`
>
> echo `date` "-- Starting HA monitor"
> while [ . ]; do
> pingresult=`ping -c 3 -W 1 $HA_Node_IP | grep time= | wc -l`
>
> if [ "$pingresult" == "0" ]; then
> echo `date` "-- HA heartbeat failed, taking over VIP"
>
> /opt/aws/bin/ec2-assign-**private-ip-addresses -n $ENI_ID
> --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
> if [ "$pingresult" == "0" ]; then
> echo `date` "-- Restarting network"
> /sbin/service network restart > /dev/null 2>&1
> fi
> sleep 60
> fi
> sleep 2
> done
>
>
>
> David Lang
>
> ______________________________**_________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/**mailman/listinfo/pacemaker
> <http://oss.clusterlabs.org/mailman/listinfo/pacemaker><http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf
> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf><http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
> Bugs: http://bugs.clusterlabs.org
>
>
>
>
> _______________________________________________
>
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
>
>
> Project Home: http://www.clusterlabs.org
>
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>
> Bugs: http://bugs.clusterlabs.org
>
>
>
> _______________________________________________
> Pacemaker mailing list:
> Pacemaker at oss.clusterlabs.orghttp://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
_______________________________________________

Pacemaker mailing list: Pacemaker at oss.clusterlabs.org

http://oss.clusterlabs.org/mailman/listinfo/pacemaker



Project Home: http://www.clusterlabs.org

Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list