[Pacemaker] Custom event script during master/slave failover?

Tue Nov 13 14:50:11 UTC 2012

David and Jake,

Thanks for the replies, that's exactly what I needed.  I think where I was
going wrong was playing with the role="Master" and role="Slave" parameters,
but always having the same timeout for both.  (crm_verify -L confused me
even more on what I was doing wrong)

Speaking of timeouts, why should I stay above 20 seconds for a monitoring
interval?  If the service crashes for whatever reason, I'd like to avoid
downtime and keep the failover window as quick as possible.  If this makes
any difference, Redis and Memcache are different beasts compared to
databases.  The clients are usually done with their requests after 20
milliseconds.  It's very fast paced, and 20 seconds of downtime is pretty
significant in this application.

Also David, after increasing my interval to 20 seconds and doing a separate
monitor definition for Master and Slave, both checks still run duplicate
monitor ops.  (See below)  Probably not a big deal, but I was just curious
on why Pacemaker would do that.

Thanks!

--Cal

------------------------------------------------------------------------------------------------------------------------------------------
redis(redis:0)[3600]:   2012/11/13_08:40:39 INFO: redis-server is running
(PID 15221)
redis(redis:0)[3600]:   2012/11/13_08:40:39 INFO: redis-server is running
(PID 15221)
redis(redis:0)[3600]:   2012/11/13_08:40:39 INFO: redis-server is running
(as master)
redis(redis:0)[3600]:   2012/11/13_08:40:39 INFO: redis-server is running
(as master)
redis(redis:0)[4175]:   2012/11/13_08:40:59 INFO: redis-server is running
(PID 15221)
redis(redis:0)[4175]:   2012/11/13_08:40:59 INFO: redis-server is running
(PID 15221)
redis(redis:0)[4175]:   2012/11/13_08:40:59 INFO: redis-server is running
(as master)
redis(redis:0)[4175]:   2012/11/13_08:40:59 INFO: redis-server is running
(as master)
------------------------------------------------------------------------------------------------------------------------------------------
redis(redis:1)[7690]:   2012/11/13_08:31:06 INFO: redis-server is running
(PID 7330)
redis(redis:1)[7690]:   2012/11/13_08:31:06 INFO: redis-server is running
(PID 7330)
redis(redis:1)[7690]:   2012/11/13_08:31:06 INFO: redis-server is running
(as slave)
redis(redis:1)[7690]:   2012/11/13_08:31:06 INFO: redis-server is running
(as slave)
redis(redis:1)[7729]:   2012/11/13_08:31:28 INFO: redis-server is running
(PID 7330)
redis(redis:1)[7729]:   2012/11/13_08:31:28 INFO: redis-server is running
(PID 7330)
redis(redis:1)[7729]:   2012/11/13_08:31:28 INFO: redis-server is running
(as slave)
redis(redis:1)[7729]:   2012/11/13_08:31:28 INFO: redis-server is running
(as slave)
------------------------------------------------------------------------------------------------------------------------------------------

On Mon, Nov 12, 2012 at 3:37 PM, David Vossel <dvossel at redhat.com> wrote:

> ----- Original Message -----
> > From: "Cal Heldenbrand" <cal at fbsdata.com>
> > To: "The Pacemaker cluster resource manager" <
> pacemaker at oss.clusterlabs.org>
> > Sent: Monday, November 12, 2012 3:27:02 PM
> > Subject: Re: [Pacemaker] Custom event script during master/slave
> failover?
> >
> >
> > I've managed to get a test Redis configuration up and running...
> > however this is the first time I've played with a master/slave
> > resource, and I'm noticing some odd behavior in the monitoring
> > actions. Setting in a super quick monitoring interval of 1 second, I
> > noticed at first only the slave was being monitored, 2x in each
> > second. Then after changing a bunch of config options and several
> > cluster restarts, I noticed the monitoring action had moved to the
> > master, again 2x events in each second.
>
> Use a longer timeout than 1s. Don't go under 20s... that might be why you
> see 2x events.
>
> The monitor action didn't move to the master or move to the slave.  There
> is a separate monitor operation used for resources in the master and slave
> role.  Both monitor operations have to be defined, looking at your config
> it appears only the Master monitor op is defined.
>
>
> http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_monitoring_multi_state_resources.html
>
> If you are new to multi-state resources. I'd recommend playing around with
> the ocf:pacemaker:Stateful resource some.  Stateful is a fake resource that
> does nothing except allow you to start/stop/promote/demote/monitor it.  It
> is useful for testing out configurations and understanding expected
> behavior.
>
> -- Vossel
>
>
>
> >
> > redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is
> > running (PID 30759)
> > redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is
> > running (PID 30759)
> > redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is
> > running (as master)
> > redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is
> > running (as master)
> >
> > Here is my primitive and master/slave definition:
> >
> > primitive redis ocf:heartbeat:redis \
> > meta target-role="Master" is-managed="true" \
> > op monitor interval="1s" role="Master" timeout="5s" on-fail="restart"
> > ms redis_clone redis \
> > meta notify="true" is-managed="true" ordered="false"
> > interleave="false" globally-unique="false" target-role="Master"
> > migration-threshold="1"
> >
> > I'd like to figure out how to make sure both master and slave are
> > actively monitored. Let me know if you need any more info on my
> > config!
> >
> > Thank you,
> >
> > --Cal
> >
> >
> >
> > On Fri, Nov 9, 2012 at 4:52 PM, David Vossel < dvossel at redhat.com >
> > wrote:
> >
> >
> >
> >
> >
> > ----- Original Message -----
> > > From: "Cal Heldenbrand" < cal at fbsdata.com >
> > > To: "The Pacemaker cluster resource manager" <
> > > pacemaker at oss.clusterlabs.org >
> > > Sent: Friday, November 9, 2012 4:29:05 PM
> > > Subject: [Pacemaker] Custom event script during master/slave
> > > failover?
> > >
> > >
> > > Hi everyone,
> > >
> > > I'm playing around with the possibility of using Pacemaker in a
> > > Redis
> > > master/slave cluster. The difficult part of this, is that Redis
> > > will
> > > not automatically flip itself from read-only slave mode into master
> > > mode. A client needs to connect to the slave server and run this
> > > SLAVEOF NO ONE command to turn it into a master.
> >
> > The resource agent should be able to do this. Pacemaker will tell the
> > resource agent to promote/demote a resource. Your Redis agent can
> > then go off and do whatever logic is necessary to actually
> > transition the process a master/slave process.
> >
> >
> > >
> > > If I set up a master/slave primitive, is it possible to fire an
> > > event
> > > script during a failover event that would allow me to do this?
> > >
> > > After the stonith event fires and the broken master reboots,
> > > I would
> > > write in something special in the Redis init.d script to detect if
> > > my peer is already a master, and start up in slave mode.
> >
> > If you are using Pacemaker, it should be controlling what node has
> > the master resource and what node has the slave resource. Take a
> > look at what OCF scripts are and how pacemaker uses them.
> >
> >
> > >
> > > Has anyone else worked up a cluster config with Redis?
> >
> > Using google, found some work someone did on a redis resource agent.
> >
> https://github.com/martinwalter/resource-agents/blob/master/heartbeat/redis
> >
> > -- Vossel
> > > Thank you,
> > >
> > > --Cal
> > >
> > > _______________________________________________
> > > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started:
> > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20121113/5ff38670/attachment.htm>