[Pacemaker] resource with colocation rule doesn't fail

[Andrew Beekhof]

On 02/08/2013, at 5:56 PM, Johan Huysmans <johan.huysmans at inuits.be> wrote:

> Hi Andrew,
> 
> Thanks for the fix.
> I tried it on my setup and now when a cloned resource fails the group will move to the other node as expected.
> 
> However I noticed something strange.
> If a cloned resource is failing I see this in the logs:
> pengine[12178]:  warning: unpack_rsc_op: Processing failed op monitor for d_bird:1 on DEM-2: not running (7)
> 
> If that same cloned resource is recovered I still see that same message appear in the logfile.
> But crm_mon shows it correctly and it functions correctly.
> 
> However when I restart the other node (or restart only the pacemaker service) it reappears as failed in the crm_mon and the cluster behaves as it is failing, however it isn't.

It comes down to this:

# PCMK_trace_functions=unpack_rsc_op tools/crm_mon -x pcmk-vr-02-aug-2013/DEM-1/pengine/pe-input-8.bz2 -V 2>&1 | grep -v -e d_bird_subnet_state -e d_bird6 | grep "Unpacking task.*d_bird.*DEM-2"

(    unpack.c:2100  )   trace: unpack_rsc_op: 	Unpacking task d_bird_last_0/start (call_id=41, status=0, rc=0, time=1375428240) on DEM-2 (role=Unknown)
(    unpack.c:2100  )   trace: unpack_rsc_op: 	Unpacking task d_bird_last_failure_0/monitor (call_id=51, status=0, rc=7, time=1375428590) on DEM-2 (role=Started)
(    unpack.c:2100  )   trace: unpack_rsc_op: 	Unpacking task d_bird_monitor_10000/monitor (call_id=51, status=0, rc=0, time=1375428811) on DEM-2 (role=Started)

vs.

# PCMK_trace_functions=unpack_rsc_op tools/crm_mon -x pcmk-vr-02-aug-2013/DEM-2/pengine/pe-input-0.bz2 -V 2>&1 | grep -v -e d_bird_subnet_state -e d_bird6 | grep "Unpacking task.*d_bird.*DEM-2" 

(    unpack.c:2100  )   trace: unpack_rsc_op: 	Unpacking task d_bird_last_0/start (call_id=41, status=0, rc=0, time=1375428240) on DEM-2 (role=Unknown)
(    unpack.c:2100  )   trace: unpack_rsc_op: 	Unpacking task d_bird_monitor_10000/monitor (call_id=51, status=0, rc=0, time=1375428240) on DEM-2 (role=Started)
(    unpack.c:2100  )   trace: unpack_rsc_op: 	Unpacking task d_bird_last_failure_0/monitor (call_id=51, status=0, rc=7, time=1375428590) on DEM-2 (role=Started)

Note the value of 'time' for d_bird_monitor_10000 in the two cases.

Now I just need to figure out why the value '1375428811' got lost.

> I have to perform a "crm resource cleanup <resource>" to clear this behaviour.
> 
> I captured this in the attached crm_report.
> 
> gr.
> Johan
> 
> On 02-08-13 05:14, Andrew Beekhof wrote:
>> On 02/08/2013, at 11:42 AM, Andrew Beekhof <andrew at beekhof.net> wrote:
>> 
>>> On 02/08/2013, at 11:33 AM, Andrew Beekhof <andrew at beekhof.net> wrote:
>>> 
>>>> On 01/08/2013, at 5:38 PM, Johan Huysmans <johan.huysmans at inuits.be> wrote:
>>>> 
>>>>> I forgot to mention:
>>>>> 
>>>>> I'm using a build from git (Version: 1.1.11-1.el6-42f2063).
>>>>> I used the same config on an old 1.1.10 rc (rc6 or before) and that worked, as of rc7 it didn't work anymore.
>>>> I will have a look, but why are you setting on-fail=block for everything?
>>> Ironically the log message for the commit which broke this was:
>>> 
>>> commit faa883cf7927d84f61f29211fe6e2980de645620
>>>    Bug: cl#5170 - Correctly support on-fail=block for clones
>> Fixed in:
>> 
>> 	https://github.com/beekhof/pacemaker/commit/66a3ea6
>> 
>> + Andrew Beekhof (2 minutes ago) 66a3ea6: Fix: PE: Do not allow colocation with blocked clone instances  (HEAD, master)
>> + Andrew Beekhof (21 minutes ago) b2c105b: Fix: PE: Do not re-allocate clone instances that are blocked in the Stopped state
>> 
>> 
>> 
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> <pcmk-vr-02-aug-2013.tar.bz2>_______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org