[Pacemaker] Why monitor fails in my RA

[Andrew Beekhof]

On Thu, Apr 26, 2012 at 6:41 AM, Greg <test123 at implix.com> wrote:
>
> Hi,
>
> I try to write redis resources agent working in master-slave. My
> configuration:
> node s1
> node s2
> primitive ip-redis ocf:heartbeat:IPaddr2 \
>        params ip="192.168.1.15" nic="eth0" cidr_netmask="24" \
>        op monitor interval="10s" timeout="30s" \
>        meta target-role="Started"
> primitive redis-server ocf:implix:redis4 \
>        op start interval="0" timeout="60s" \
>        op stop interval="0" timeout="60s" \
>        op monitor interval="5s" role="Master" timeout="60s" \
>        op monitor interval="10s" role="Slave" timeout="60s" \
>        params masterip="192.168.1.15"
> ms redis-ms redis-server \
>        meta master-max="1" master-node-max="1" clone-max="2" \
>        clone-node-max="1" target-role="Master"
> colocation co-redis-ms inf: ip-redis redis-ms:Master
> order or-redis inf: redis-ms:promote ip-redis:start
> property $id="cib-bootstrap-options" \
>        dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
>        cluster-infrastructure="openais" \
>        no-quorum-policy="ignore" \
>        stonith-enabled="false" \
>        expected-quorum-votes="2" \
>        default-action-timeout="20s" \
>        last-lrm-refresh="1335271825" \
>        default-resource-stickiness="10"
>
>
>        To simplify RA all redis nodes start as a slave (that's why I need to
> pass masterip in configuration).
>
>        Script works great it promote on secondary (if master node is down)
> but only few times. In some point sometimes after 2 or after 3 master fails
> (manually kill process) I get this error:
> redis-server:0_monitor_5000 (node=s1, call=16, rc=9, status=complete):
> master (failed)
>
> My mointor function (simplified and removed overhead and added some
> comments) is:
> redis_monitor() {
>        # I set score 10 for master 5 is for slave
>        CURSCORE=`$CRM_MASTER -G -q`
>        logger "redis_monitor: score $CURSCORE"
>        local state
>        redis_state
>
>        # In RET is current local redis state
>        state=$(echo "${RET}" | cut -d':' -f2 | tr -d '\r')
>
>        if [ "${state}" = "master" ];then
>                $CRM_MASTER -v $CRM_MASTER_SCORE # score is 10
>                exit $OCF_RUNNING_MASTER
>        fi
>
>        if [ "${state}" = "slave" ];then
>                $CRM_MASTER -v $CRM_SLAVE_SCORE # score is 5
>                exit $OCF_SUCCESS
>        fi
>
>        # if not slave/master so resource is failed
>        $CRM_MASTER -l reboot -D
>        if [ $CURSCORE -eq $CRM_MASTER_SCORE ];then
>                exit $OCF_FAILED_MASTER
>        fi
>
>        exit $OCF_NOT_RUNNING

Are you sure its NOT_RUNNING?
Could it also be running but generally failed?

> }
>
> From my logs I know that monitoring function returned OCF_FAILED_MASTER when
> master is down and then this error occurred:
> redis-server:0_monitor_5000 (node=s1, call=16, rc=9, status=complete):
> master (failed)
>
> After that failed master node is not monitored on that node until I run
> cleanup:
> #crm resource cleanup redis-server:0
>
>
> My questions:
> 1) What I'm doing wrong ?. How can I fix this.
> I've tried on-fail="restart" but this not helped

You'd need to supply more information (in the form of a hb_report tarball).
An upgrade might not hurt either.

>
> 2) Using older version of redis 2.3 If master failed redis is hanging for
> some time (21-24 seconds). Even I set higher timeout on monitor functions it
> still timeout after 20 seconds why?.

How did you set the timeout higher?

> (Changing default-action-timeout to higher value helped to resolve this but
> I think timeout should be enough)
>
>
>
> --
> Greg
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org