[Pacemaker] Help with N+1 configuration

[Cal Heldenbrand]

A few more questions, as I test various outage scenarios:

My memcached OCF script appears to give a false positive occasionally, and
pacemaker restarts the service.  Under the hood, it uses netcat to
localhost with a 3 second connection timeout.  I've run my script manually
in a loop and it never seems to time out.

My primitive looks like this:

primitive memcached ocf:fbs:memcached \
        meta is-managed="true" target-role="Started" \
        op monitor interval="1s" timeout="5s"

I've played around with the primitive's interval and timeout.  All that
seems to do is decrease the frequency that the false positive happens.  Is
there any way to add logic to the monitor to say "restart the service only
if 3 failures in a row happen?"

Also, I've tried to create a massive load failure by using a fork bomb.  A
few of the outages we've had on our memcache servers appear to be heavy
loads -- the machine response to ICMP on the ethernet card, but doesn't
respond on ssh.  A fork bomb pretty much recreates the same problem.  When
I fire off a fork bomb on my test machine, it seems to take 5 minutes or
more to actually trigger the failover event.  It's difficult for me to make
sense of all the logging going on, but these two timeout values seem to be
interesting:

crmd:    error: crm_timer_popped:         Election Timeout (I_ELECTION_DC)
just popped in state S_ELECTION! (120000ms)
crmd:    error: crm_timer_popped:         Integration Timer (I_INTEGRATED)
just popped in state S_INTEGRATION! (180000ms)

Can those values be adjusted?  Or is there a common configuration change to
be more responsive to an active content check like I'm doing?

For reference, please see my attached script *memcached*.

Thanks!

--Cal

On Thu, Jul 26, 2012 at 1:35 PM, Phil Frost <phil at macprofessionals.com>wrote:

> On 07/26/2012 02:16 PM, Cal Heldenbrand wrote:
>
>> That seems very handy -- and I don't need to specify 3 clones?   Once my
>> memcached OCF script reports a downed service, one of them will
>> automatically transition to the current failover node?
>>
>
> There are options for the clone on how many instances of the cloned
> resource to create, but they default to the number of nodes in the cluster.
> See: http://www.clusterlabs.org/**doc/en-US/Pacemaker/1.1/html/**
> Pacemaker_Explained/**ch10s02s02.html<http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ch10s02s02.html>
>
>
>  Is there any reason you specified just a single memcache_clone, instead
>> of both the memcache primitive and memcached_clone?  I might not be
>> understanding exactly how a clone works.  Is it like... maybe a "symbolic
>> link" to a primitive, with the ability to specify different metadata and
>> parameters?
>>
>
> Once you make a clone, the underlying primitive isn't referenced anywhere
> else (that I can think of). If you want to stop memcache, you don't stop
> the primitive; you add a location constraint forbidding the clone from
> running on the node where you want to stop memcache ("crm resource migrate"
> is easiest). I can't find the relevant documentation, but this is just how
> they work. The same is true for groups -- the member primitives are never
> referenced except by the group. I believe in most cases if you try to
> reference the primitive, you will get an error.
>
>
>  Despite the advertisement of consistent hashing with memcache clients,
>> I've found that they still have long timeouts waiting on connecting to an
>> IP.  So, keeping the clustered IPs up at all times is more important than
>> having a seasoned cache behind them.
>>
>
> I don't know a whole lot about memcache, but it sounds like you might even
> want to reduce the colocation score for the ips on memcache to be a large
> number, but not infinity. This way in the case that memcache is broken
> everywhere, the ips are still permitted to run. This might also cover you
> in the case that a bug in your resource agent thinks memcache has failed
> everywhere, but actually it's still running fine. The decision depends
> which failure the memcache clients handle better: the IP being down, or the
> IP being up but not having a working memcache server behind it.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20120726/9bc0c847/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memcached
Type: application/octet-stream
Size: 6089 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20120726/9bc0c847/attachment-0004.obj>