[Pacemaker] custom resource-agent

Wed Oct 2 08:28:03 UTC 2013

the changes take instant effect, i let "watch
/usr/lib64/nagios/plugins/check_tcp -H my.vip.add.r -p 3306" on a 3rd
instance running, and the second i issued on the new master node the aws
command it changed from CRITICAL to OK

On Wed, Oct 2, 2013 at 4:13 PM, Peter Romfeld <peter.romfeld.hk at gmail.com>wrote:

> In you VPC the change take effect after you got "true", you can check it
> with describe-eip or in console :)
> right now i just try to add the command with hardcoded variables to
> IPaddr2 start, just to get it running for now
>
> i put my attempt to create a resource agent on github, maybe someone can
> help me find why params are not working?
> wget https://raw.github.com/peterromfeldhk/pacemaker/master/AWSFIP
>
> sorry im just started with scripting with nagios_nrpe last project :)
> i have 2 problems, 1s2 the OCF_RESKEY_ params dont work as i thought, if i
> hardcode the variables the commands are working at least in testrun, the
> second big issue i have is the monitoring
>
>
> On Wed, Oct 2, 2013 at 2:34 PM, David Lang <david at lang.hm> wrote:
>
>> Unless something has changed in the AWS API in the last few months, when
>> the aws command exits successfully, that doesn't mean the change has taken
>> effect, just that the aws system has accepted the change and it will take
>> effect 'soon'
>>
>>
>> David Lang
>>
>> On Wed, 2 Oct 2013, Peter Romfeld wrote:
>>
>>  yes i need to use the aws command, i am using a VPC, after issueing the
>>> command i get a "true" statement and its done
>>>
>>> so i only want pacemaker to issue the one-shot command at failover. Here
>>> is
>>> what i have atm: (i know its still dirty, just lerning pacemaker)
>>>
>>> primitive drbd_mysql ocf:linbit:drbd \
>>>        params drbd_resource="mydata" \
>>>        op monitor interval="15s"
>>> primitive fs_mysql ocf:heartbeat:Filesystem \
>>>        params device="/dev/drbdx" directory="/mountpint" fstype="ext4"
>>> options="relatime,barrier=1" \
>>>        op start interval="0" timeout="60" \
>>>        op stop interval="0" timeout="60" \
>>>        op monitor interval="10s" timeout="60s" OCF_CHECK_LEVEL="20" \
>>>        meta target-role="started"
>>> primitive fvip ocf:heartbeat:AWSFIP \
>>>        params fvip="192.168.2.10" region="ap-southeast-1"
>>> primitive ip_mysql ocf:heartbeat:IPaddr2 \
>>>        params ip="192.168.2.10" cidr_netmask="20" \
>>>        op monitor interval="10" \
>>>        meta target-role="started"
>>> primitive mysqld lsb:mysql
>>> group mysql fs_mysql ip_mysql mysqld
>>> ms ms_drbd_mysql drbd_mysql \
>>>        meta master-max="1" master-node-max="1" clone-max="2"
>>> clone-node-max="1" notify="true"
>>> colocation mysql_on_drbd inf: fvip mysql ms_drbd_mysql:Master
>>> order mysql_after_drbd_and_fvip inf: ms_drbd_mysql:promote fvip:start
>>> mysql:start
>>>
>>> my AWSFIP(adjusted Dummy :%s/dummy/awsfip/g|%s/Dummy/**AWSFIP/g):
>>>
>>> <parameter name="fvip" unique="1" required="1">
>>> <longdesc lang="en">
>>> The IPv4 address to be configured in dotted quad notation, for example
>>> "192.168.1.1".
>>> </longdesc>
>>> <shortdesc lang="en">IPv4 address</shortdesc>
>>> <content type="string" default="" />
>>> </parameter>
>>>
>>> <parameter name="region" unique="1" required="1">
>>> <longdesc lang="en">
>>> The name of the AWS region
>>> </longdesc>
>>> <shortdesc lang="en">AWS region</shortdesc>
>>> <content type="string"/>
>>> </parameter>
>>>
>>> awsfip_start() {
>>>    awsfip_monitor
>>>    Instance_ID=`/usr/bin/curl --silent
>>> http://169.254.169.254/latest/**meta-data/instance-id`<http://169.254.169.254/latest/meta-data/instance-id>
>>>    ENI_ID=`aws ec2 describe-instances --instance-id $Instance_ID --region
>>> $OCF_RESKEY_region | grep NetworkInterfaceId | cut -d '"' -f 4`
>>>    if [ $? =  $OCF_SUCCESS ]; then
>>>        return $OCF_SUCCESS
>>>    fi
>>>         aws ec2 assign-private-ip-addresses --network-interface-id
>>> $ENI_ID
>>> --private-ip-addresses $OCF_RESKEY_fvip --allow-reassignment --region
>>> $OCF_RESKEY_region
>>>         sleep 4
>>>         aws ec2 assign-private-ip-addresses --network-interface-id
>>> $ENI_ID
>>> --private-ip-addresses $OCF_RESKEY_fvip --allow-reassignment --region
>>> $OCF_RESKEY_region
>>>         /etc/init.d/networking restart
>>>    touch ${OCF_RESKEY_state}
>>> }
>>>
>>> I couldn't get it to work yet, and i don't want to run a external script
>>> for it. I can't be so hard to let pacemaker execute an additional
>>> one-shot
>>> command at failover (in the correct order..)
>>>
>>> Thanks for your help!
>>>
>>>
>>> On Wednesday, October 02, 2013 07:33 AM, David Lang wrote:
>>>
>>> the aws command is making the call to inform aws, if you were to bring up
>>> the address without making the aws command, would it work? If you are on
>>> a
>>> Virtual Private Cloud (VPC), it may, but I didn't think it would.
>>>
>>> If you can make it work without the aws command, then you can just use
>>> the
>>> standard pacemaker VIP configuration. I know that this doesn't work if
>>> you
>>> have an external IP that you are moving (you must use an aws call to tell
>>> Amazon to move the IP), but it's possible that you don't have to for an
>>> internal IP, but I would be surprised.
>>>
>>> David Lang
>>>
>>>
>>> On Wed, 2 Oct 2013, Peter Romfeld wrote:
>>>
>>> Hey,
>>> when i change the secondary IP per hand or with external script on a
>>> Ubuntu
>>> Instance I just need:
>>> /etc/network/interfaces
>>> auto eth0
>>> iface eth0 inet dhcp
>>>   address 192.168.32.12
>>>   netmask 255.255.240.0
>>>   gateway 192.168.32.1
>>>   up ip addr add 192.168.32.11/20 dev eth0
>>>
>>> and then run the script which basically just does:
>>> #!/bin/sh
>>>
>>> VIP=172.32.32.11
>>> REGION=ap-southeast-1
>>>
>>> Instance_ID=`/usr/bin/curl --silent
>>> http://169.254.169.254/latest/**meta-data/instance-id`<http://169.254.169.254/latest/meta-data/instance-id>
>>> ENI_ID=`aws ec2 describe-instances --instance-id $Instance_ID --region
>>> $REGION | grep NetworkInterfaceId | cut -d '"' -f 4`
>>>
>>> aws ec2 assign-private-ip-addresses --network-interface-id $ENI_ID
>>> --private-ip-addresses $VIP --allow-reassignment --region $REGION
>>>
>>>
>>> I dont need to inform AWS or restart network, only the correct network
>>> config and the one command, when i tested it with pinging from a 3rd
>>> instance during IP change i didnt got any interupts. I dont know about
>>> monitoring it
>>>
>>>
>>> On Wed, Oct 2, 2013 at 1:38 AM, David Lang <david at lang.hm>
>>> <david at lang.hm>wrote:
>>>
>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>
>>> On Tue, Oct 01, 2013 at 10:07:12AM -0700, David Lang wrote:
>>>
>>>
>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>
>>> On Tue, Oct 01, 2013 at 07:22:20AM -0700, David Lang wrote:
>>>
>>>
>>> On Tue, 1 Oct 2013, Dejan Muhamedagic wrote:
>>>
>>> Hi David,
>>>
>>>
>>> On Mon, Sep 30, 2013 at 12:41:23PM -0700, David Lang wrote:
>>>
>>> On Mon, 30 Sep 2013, David Lang wrote:
>>>
>>> On Mon, 30 Sep 2013, Michael Schwartzkopff wrote:
>>>
>>>
>>> Am Montag, 30. September 2013, 21:12:56 schrieb Peter Romfeld:
>>>
>>>
>>> I am working in AWS i cant just use a VIP i need to use a floating
>>> secondary IP which i reassign through script, i want to let
>>> pacemaker
>>> handle the reassignment...
>>>
>>>
>>> Please explain the difference of a VIP and a "secondary IP" in
>>> your opinion.
>>>
>>>
>>> with AWS you need to inform amazon of the change, not just change
>>> the IP on the local box, that requires much more work than a
>>> simple local VIP
>>>
>>>
>>> being more detailed, instead of just
>>> ifconfig eth0:0 $vip
>>> you have to do something like
>>>
>>> /opt/aws/bin/ec2-assign-****private-ip-addresses -n $ENI_ID
>>> --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
>>>
>>>
>>> We may consider adding such an option to IPaddr2. Has anybody
>>> ever tried that?
>>>
>>> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
>>>
>>> if [ "$pingresult" == "0" ]; then
>>> echo `date` "-- Restarting network"
>>> /sbin/service network restart > /dev/null 2>&1
>>>
>>>
>>> That may break the cluster communication, which may lead to split
>>> brain, etc. Is that really the only way?
>>>
>>>
>>> It's not the only way, but you do have the problem that the call to
>>> aws management interface is asynchronous, you don't know when it's
>>> going to complete, and until it does, the IP doesn't actually work.
>>>
>>>
>>> Wouldn't it be then safer to wait until it starts working, i.e.
>>> to monitor in a loop?
>>>
>>>
>>> that's exactly what the snippet of code above is for, to detect when
>>> the other box no longer has the address.
>>>
>>>
>>> Hmm, perhaps I'm missing something, but I couldn't notice a loop
>>> in that code. What I meant was something like this:
>>>
>>> while ! ping -c 1 -W 1 $VIP | grep -qs time=; do
>>>        :
>>> done
>>>
>>> Then network restart wouldn't be necessary, right? Sorry, I don't
>>> know much about aws.
>>>
>>>
>>> I haven't used this exact script before, but I have seen the problem that
>>> this script is designed to address. I am not saying that I agree with
>>> this
>>> script, but it's what Amazon is suggesting, so it's probably a reasonable
>>> start.
>>>
>>>
>>> this was a cut-n-paste from the URL provided earlier
>>> http://aws.amazon.com/****articles/2127188135977316<http://aws.amazon.com/**articles/2127188135977316>
>>> <http://aws.amazon.com/**articles/2127188135977316<http://aws.amazon.com/articles/2127188135977316>
>>> ><htt**p://aws.amazon.com/articles/**2127188135977316<http://aws.amazon.com/articles/2127188135977316>
>>> >
>>>
>>>
>>>
>>> #!/bin/sh
>>> # This script will monitor another HA node and take over a Virtual IP
>>> (VIP)
>>> # if communication with the other node fails
>>>
>>> # High Availability IP variables
>>> # Other node's IP to ping and VIP to swap if other node goes down
>>> HA_Node_IP=10.0.0.11
>>> VIP=10.0.0.10
>>>
>>> # Specify the EC2 region that this will be running in
>>> REGION=us-west-2
>>>
>>> # Run aws-apitools-common.sh to set up default environment variables and
>>> to
>>> # leverage AWS security credentials provided by EC2 roles
>>> . /etc/profile.d/aws-apitools-****common.sh
>>>
>>> # Determine the instance and ENI IDs so we can reassign the VIP to the
>>> # correct ENI. Requires EC2 describe-instances and
>>> assign-private-ip-address
>>> # permissions. The following example EC2 roles policy will authorize
>>> these
>>> # commands:
>>> # {
>>> # "Statement": [
>>> # {
>>> # "Action": [
>>> # "ec2:AssignPrivateIpAddresses"****,
>>> # "ec2:DescribeInstances"
>>> # ],
>>> # "Effect": "Allow",
>>> # "Resource": "*"
>>> # }
>>> # ]
>>> # }
>>>
>>> Instance_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/****<http://169.254.169.254/latest/**>
>>> meta-data/instance-id`ENI_ID=`****/opt/aws/bin/ec2-describe-****
>>> instances
>>> <http://169.254.169.254/**latest/meta-data/instance-**
>>> idENI_ID=/opt/aws/bin/ec2-**describe-instances<http://169.254.169.254/latest/meta-data/instance-idENI_ID=/opt/aws/bin/ec2-describe-instances>
>>> ><http://**169.254.169.254/latest/meta-**data/instance-idENI_ID=/opt/**
>>> aws/bin/ec2-describe-instances<http://169.254.169.254/latest/meta-data/instance-idENI_ID=/opt/aws/bin/ec2-describe-instances>
>>> **>$Instance_ID
>>>
>>> --region $REGION | grep eni -m 1 | awk '{print $2;}'`
>>>
>>> echo `date` "-- Starting HA monitor"
>>> while [ . ]; do
>>> pingresult=`ping -c 3 -W 1 $HA_Node_IP | grep time= | wc -l`
>>>
>>> if [ "$pingresult" == "0" ]; then
>>> echo `date` "-- HA heartbeat failed, taking over VIP"
>>>
>>> /opt/aws/bin/ec2-assign-****private-ip-addresses -n $ENI_ID
>>> --secondary-private-ip-address $VIP --allow-reassignment --region $REGION
>>> pingresult=`ping -c 1 -W 1 $VIP | grep time= | wc -l`
>>> if [ "$pingresult" == "0" ]; then
>>> echo `date` "-- Restarting network"
>>> /sbin/service network restart > /dev/null 2>&1
>>> fi
>>> sleep 60
>>> fi
>>> sleep 2
>>> done
>>>
>>>
>>>
>>> David Lang
>>>
>>> ______________________________****_________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/****mailman/listinfo/pacemaker<http://oss.clusterlabs.org/**mailman/listinfo/pacemaker>
>>> <http://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>> ><ht**tp://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>> >
>>>
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/****
>>> doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf>
>>> <http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>> ><**http://www.clusterlabs.org/**doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>> >
>>>
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>>
>>>
>>> ______________________________**_________________
>>>
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/**mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>>
>>>
>>>
>>> Project Home: http://www.clusterlabs.org
>>>
>>> Getting started: http://www.clusterlabs.org/**
>>> doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>>
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>>
>>> ______________________________**_________________
>>> Pacemaker mailing list:
>>> Pacemaker at oss.clusterlabs.**orghttp://oss.clusterlabs.org/**
>>> mailman/listinfo/pacemaker<http://oss.clusterlabs.org/mailman/listinfo/pacemaker>
>>>
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/**
>>> doc/Cluster_from_Scratch.pdf<http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>> _______________________________________________
>>
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>>
>>
>> Project Home: http://www.clusterlabs.org
>>
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20131002/c444dd84/attachment.htm>