[Pacemaker] Automating Pacemaker Setup

[veghead]

Todd Nine <todd at ...> writes:

>   I have a setup nearly working.  Would you be willing to share recipes?
...
> It's not quite working yet, but it's close.  Since you've managed to get
> this working, you may be able to finish these off.  I have everything
> working except the init start/stop hooks for pacemaker to set the
> Elastic IP automatically and then run chef-client to reconfigure
> everything on all the other nodes.

Wow. The example pacemaker config and the trick of starting heartbeat before 
using crm configure were the last steps I needed. Thanks!

So, here's how I got Elastic IP failover working. I can't claim credit for the 
idea... I found a basic example here: https://forums.aws.amazon.com/thread.jspa?
messageID=195373. That didn't quite work for me, so I rewrote the LSB script in 
pure ruby and leveraged the amazon-ec2 gem (https://github.com/grempe/amazon-
ec2) to handle associating the EIP with the current instance. I have included my 
script below. A couple of key things. 

First, I found that when an instance loses it's elastic ip (whether through 
"disassociate" or another instance grabbed eip), it loses public internet 
connectivity for 1-3 minutes. Apparently this is expected, according to AWS 
Support: https://forums.aws.amazon.com/message.jspa?messageID=250571#250571. As 
a result, I decided it didn't make any sense to have the "stop" method for the 
EIP LSB script do anything.

Second, my Pacemaker configure is pretty close to yours. I setup the nodes with 
ucast in almost the exact same manner. The key differences are all in setting up 
the primitives with the correct order and colocation:

primitive elastic_ip lsb:elastic-ip op monitor interval="10s"
primitive haproxy lsb:haproxy op monitor interval="10s"
order haproxy-after-eip inf: elastic_ip haproxy
colocation haproxy-with-eip inf: haproxy elastic_ip

Third, here's my elastic-ip.rb LSB script that handles the Elastic IP. Since LSB 
scripts can't take any parameters other than the usual start/stop/status/etc, I 
treat the script as a Chef template and inject the desired EIP into the 
template. The other secret is that I created a special user using AWS IAM with a 
policy that only allows the user to associate/disassociate EIP addresses. I 
store the AccessKey and SecretAccessKey in a file in /etc/aws/pacemaker_keys. 

Let me know if you have any questions. And thanks for the tip on using crm to 
initialize pacemaker.

#!/usr/bin/ruby

# Follows the LSB Spec: http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-
generic/LSB-Core-generic/iniscrptact.html

require 'rubygems'
require 'AWS'

ELASTIC_IP="<%= @elastic_ip %>"
EC2_INSTANCE_ID=`wget -T 5 -q -O - http://169.254.169.254/latest/meta-
data/instance-id`

# Load the AWS access keys
properties = {}
File.open("/etc/aws/pacemaker_keys", 'r') do |file|
  file.read.each_line do |line|
    line.strip!
    if (line[0] != ?# and line[0] != ?=)
      i = line.index('=')
      if (i)
        properties[line[0..i - 1].strip] = line[i + 1..-1].strip
      else
        properties[line] = ''
      end
    end
  end
end
AWS_ACCESS_KEY = properties["AWS_ACCESS_KEY"].delete "\""
AWS_SECRET_ACCESS_KEY = properties["AWS_SECRET_ACCESS_KEY"].strip.delete "\""

[ ELASTIC_IP, EC2_INSTANCE_ID, AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY ].each do 
|value|
  if value.nil? || value.length == 0
    exit case ARGV[0]
      when "status" then 4
      else 1
    end
  end
end

def status(ec2)
  # Typical responses look like the following:
  #   {"requestId"=>"065d1661-31b1-455d-8f63-ba086b8104de", "addressesSet"=>
{"item"=>[{"instanceId"=>"i-22e93a4d", "publicIp"=>"50.19.93.215"}]}, 
"xmlns"=>"http://ec2.amazonaws.com/doc/2010-08-31/"}
  # or
  #   {"requestId"=>"9cd3ab7e-1c03-4821-9565-1791dd1bb0fc", "addressesSet"=>
{"item"=>[{"instanceId"=>nil, "publicIp"=>"174.129.34.161"}]}, 
"xmlns"=>"http://ec2.amazonaws.com/doc/2010-08-31/"}
  response = ec2.describe_addresses({:public_ip => ELASTIC_IP})
  retval = 4
  if ! response.nil?
    if ! response["addressesSet"].nil?
      if ! response["addressesSet"]["item"].nil? && response["addressesSet"]
["item"].length >= 1
        if response["addressesSet"]["item"][0]["instanceId"] == EC2_INSTANCE_ID
          retval = 0
        else
          retval = 3
        end
      end
    end
  end
  retval
end

def start(ec2)
  # Throws exception if the instance does not exist or the address does not 
belong to us
  retval = 1
  begin
    response = ec2.associate_address({ :public_ip => ELASTIC_IP, :instance_id => 
EC2_INSTANCE_ID })
    retval = 0
  rescue => e
    puts "Error attempting to associate address: " + e
  end
  retval
end

def stop(ec2)
  0
end

def reload(ec2)
  start(ec2)
end

def force_reload(ec2)
  reload(ec2)
end

def restart(ec2)
  start(ec2)
end

def try_restart(ec2)
  start(ec2)
end

ec2 = AWS::EC2::Base.new(:access_key_id => AWS_ACCESS_KEY, :secret_access_key => 
AWS_SECRET_ACCESS_KEY)

retval = case ARGV[0]
  when "status" then status(ec2)
  when "start" then start(ec2)
  when "stop" then stop(ec2)
  when "reload" then reload(ec2)
  when "force-reload" then force_reload(ec2)
  when "restart" then restart(ec2)
  when "try-restart" then try_restart(ec2)
  else 3
end

puts "elastic-ip " + (retval == 0 ? "OK" : "FAIL")
exit retval