[ClusterLabs] Pacemaker resource is not tried to be recovered after failure on slave node even when failcount is less than migration-threshold

Tue Nov 28 00:05:06 EST 2017

Отправлено с iPhone

> 27 нояб. 2017 г., в 14:50, Pankaj <pankaj386 at gmail.com> написал(а):
> 
> Hi,
> 
> Could you please help me with below query.
> 
> I have a stateful resource, stateful_ms, defined as below. The migration-threshold is defined as 4 and resource-stickiness as 100.
> I have pacemaker cluster of 5 nodes. First time resource(stateful_ms) is up as MASTER on node-0 and as slave in other nodes.
> I made monitor of stateful_ms to fail on node-0. As expected, after reaching failcount=4 on node-0, resource instance on node-1 is promoted to be MASTER.
> But, when monitor, of stateful_ms in node-1, is made to fail, it was seen that resource on node-0 is promoted immediately even failcount was 1 on node-1 against defined migration-threshold=4.
> 
> Could you please help me to understand:
> 1. Why resource instance on node-1 is not promoted again when its failcount is still less than migration-threshold?
> 2. How we can make sure than first resource is tried to be recovered on current node(where it failed), as per migration-threshold, then only any other node instance is promoted?
> 
> Below are the setup details:
> resource configure
> crm configure primitive stateful_dummy ocf:pacemaker:Stateful op monitor interval="5" role="Master" timeout="20" op monitor interval="10" role="Slave" timeout="20" meta resource-stickiness="100"
> crm configure ms stateful_ms stateful_dummy meta resource-stickiness="100" notify="true" master-max="1" interleave="true" migration-threshold=4 failure-timeout=60
> 

Most likely failure counter gets reset due to low failure timeout; then after master failure pacemaker first tries to demote it and ends up with two slaves which have equal master score. From here it can select any. Try with larger value for failure-timeout first.

> # crm status
> Stack: corosync
> Current DC: NODE-0 (version 1.1.16-94ff4df51a) - partition with quorum
> Last updated: Tue Nov 14 12:54:52 2017
> Last change: Tue Nov 14 12:54:43 2017 by root via cibadmin on NODE-1
> 
> 5 nodes configured
> 
>  Master/Slave Set: stateful_ms [stateful_dummy]
>      Masters: [ NODE-0 ]
>      Slaves: [ NODE-1 NODE-2 NODE-3 NODE-4 ]
> 
> 
> # crm configure show
> node 1: NODE-0
> node 2: NODE-1
> node 3: NODE-2
> node 4: NODE-3
> node 5: NODE-4
> 
> primitive stateful_dummy ocf:pacemaker:Stateful \
>         op monitor interval=5 role=Master timeout=20 \
>         op monitor interval=10 role=Slave timeout=20 \
>         meta resource-stickiness=100
> ms stateful_ms stateful_dummy \
>         meta resource-stickiness=100 notify=true master-max=1 interleave=true migration-threshold=4 failure-timeout=60 target-role=Started
>         
> 
> #crm resource score
> Allocation scores and utilization information:
> Original: NODE-0 capacity:
> Original: NODE-1 capacity:
> Original: NODE-2 capacity:
> Original: NODE-3 capacity:
> Original: NODE-4 capacity:
> clone_color: stateful_ms allocation score on NODE-0: 0
> clone_color: stateful_ms allocation score on NODE-1: 0
> clone_color: stateful_ms allocation score on NODE-2: 0
> clone_color: stateful_ms allocation score on NODE-3: 0
> clone_color: stateful_ms allocation score on NODE-4: 0
> clone_color: stateful_dummy:0 allocation score on NODE-0: 110
> clone_color: stateful_dummy:0 allocation score on NODE-1: 0
> clone_color: stateful_dummy:0 allocation score on NODE-2: 0
> clone_color: stateful_dummy:0 allocation score on NODE-3: 0
> clone_color: stateful_dummy:0 allocation score on NODE-4: 0
> clone_color: stateful_dummy:1 allocation score on NODE-0: 0
> clone_color: stateful_dummy:1 allocation score on NODE-1: 105
> clone_color: stateful_dummy:1 allocation score on NODE-2: 0
> clone_color: stateful_dummy:1 allocation score on NODE-3: 0
> clone_color: stateful_dummy:1 allocation score on NODE-4: 0
> clone_color: stateful_dummy:2 allocation score on NODE-0: 0
> clone_color: stateful_dummy:2 allocation score on NODE-1: 0
> clone_color: stateful_dummy:2 allocation score on NODE-2: 105
> clone_color: stateful_dummy:2 allocation score on NODE-3: 0
> clone_color: stateful_dummy:2 allocation score on NODE-4: 0
> clone_color: stateful_dummy:3 allocation score on NODE-0: 0
> clone_color: stateful_dummy:3 allocation score on NODE-1: 0
> clone_color: stateful_dummy:3 allocation score on NODE-2: 0
> clone_color: stateful_dummy:3 allocation score on NODE-3: 105
> clone_color: stateful_dummy:3 allocation score on NODE-4: 0
> clone_color: stateful_dummy:4 allocation score on NODE-0: 0
> clone_color: stateful_dummy:4 allocation score on NODE-1: 0
> clone_color: stateful_dummy:4 allocation score on NODE-2: 0
> clone_color: stateful_dummy:4 allocation score on NODE-3: 0
> clone_color: stateful_dummy:4 allocation score on NODE-4: 105
> native_color: stateful_dummy:2 allocation score on NODE-0: 0
> native_color: stateful_dummy:2 allocation score on NODE-1: 0
> native_color: stateful_dummy:2 allocation score on NODE-2: 105
> native_color: stateful_dummy:2 allocation score on NODE-3: 0
> native_color: stateful_dummy:2 allocation score on NODE-4: 0
> native_assign_node: stateful_dummy:2 utilization on NODE-2:
> native_color: stateful_dummy:4 allocation score on NODE-0: 0
> native_color: stateful_dummy:4 allocation score on NODE-1: 0
> native_color: stateful_dummy:4 allocation score on NODE-2: -INFINITY
> native_color: stateful_dummy:4 allocation score on NODE-3: 0
> native_color: stateful_dummy:4 allocation score on NODE-4: 105
> native_assign_node: stateful_dummy:4 utilization on NODE-4:
> native_color: stateful_dummy:1 allocation score on NODE-0: 0
> native_color: stateful_dummy:1 allocation score on NODE-1: 105
> native_color: stateful_dummy:1 allocation score on NODE-2: -INFINITY
> native_color: stateful_dummy:1 allocation score on NODE-3: 0
> native_color: stateful_dummy:1 allocation score on NODE-4: -INFINITY
> native_assign_node: stateful_dummy:1 utilization on NODE-1:
> native_color: stateful_dummy:3 allocation score on NODE-0: 0
> native_color: stateful_dummy:3 allocation score on NODE-1: -INFINITY
> native_color: stateful_dummy:3 allocation score on NODE-2: -INFINITY
> native_color: stateful_dummy:3 allocation score on NODE-3: 105
> native_color: stateful_dummy:3 allocation score on NODE-4: -INFINITY
> native_assign_node: stateful_dummy:3 utilization on NODE-3:
> native_color: stateful_dummy:0 allocation score on NODE-0: 110
> native_color: stateful_dummy:0 allocation score on NODE-1: -INFINITY
> native_color: stateful_dummy:0 allocation score on NODE-2: -INFINITY
> native_color: stateful_dummy:0 allocation score on NODE-3: -INFINITY
> native_color: stateful_dummy:0 allocation score on NODE-4: -INFINITY
> native_assign_node: stateful_dummy:0 utilization on NODE-0:
> stateful_dummy:0 promotion score on NODE-0: 10
> stateful_dummy:1 promotion score on NODE-1: 5
> stateful_dummy:2 promotion score on NODE-2: 5
> stateful_dummy:3 promotion score on NODE-3: 5
> stateful_dummy:4 promotion score on NODE-4: 5
> 
> Regards,
> Pankaj
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org