[Pacemaker] WARN: ..... unmanaged failed resources cannot prevent clone shutdown

[Andrew Beekhof]

On Wed, Jul 6, 2011 at 10:26 PM, Andreas Kurz <andreas.kurz at linbit.com> wrote:
> On 2011-07-05 00:24, Andrew Beekhof wrote:
>> On Fri, Jul 1, 2011 at 9:23 PM, Andreas Kurz <andreas.kurz at linbit.com> wrote:
>>> Hello,
>>>
>>> In a cluster without stonith enabled (yes I know ....) the monitor
>>> failure of one resource followed by the stop failure of a dependent
>>> resource lead to a cascade of errors especially because the cluster did
>>> not stop the shutdown sequence on stop (timeout) failures:
>>>
>>> WARN: should_dump_input: Ignoring requirement that
>>> resource_fs_home_stop_0 comeplete before ms_drbd_home_demote_0:
>>> unmanaged failed resources cannot prevent clone shutdown
>>>
>>> ... and that is really ugly in a DRBD Environment, because demote/stop
>>> will not work when the DRBD device is in use -- so in this case this
>>> order requirement on stop must not be ignored.
>>
>> Did you ask the cluster to shut down before or after the first resource failed?
>
> Neither the first nor the latter ...
>
> * IP resource had a monitor failure
> * restart triggered a restart of all dependent resources
> * one resource had an stop failure
> * cluster decided that this failed resource must move away
> * cluster decided to move IP to second node and therefore all dependant
> resource have to follow
> * cascading stop begins
> * two file systems where unable to umount --> stop failure/unmanaged,
> ignored
>
> ... and now the really ugly things happened as demote and stop on DRBD
> ms resources were triggered although the files system were still online.
>
> Furthermore the cluster additionally tried to promote DRBD on the second
> node which is also impossible if the other side is not demoted/stopped.
>
> Of course there are clones/ms resources that can stop independent of
> their dependent resources but DRBD is one that can't.
>
> So I think there should be a way to tell a clone/ms resource do _not_
> ignore the order requirements on stop failures of dependent resources.
>
>>
>>> The result were a lot of unmanaged resources and the cluster even tried
>>> to promote the MS resource on the other node although the second
>>> instance was neither demoted nor stopped.
>>
>> We seem to loose either way.
>> If we have the cluster block people complain shutdown takes too long.
>
> This seems to be sensible for a cluster shutdown, but I don't think this
> a good behavior on resource migration. The cluster was really in a heavy
> mess after this stop cascade.

Which was caused by a failed stop ;-)
None of this would have happened if stonith was enabled or if the
original stop was reliable (which it is required to be).

>
> I would expect the cluster to block on the first stop error and wait for
> manual intervention if no fencing is configured.

I think you'd need to set on-fail=block for that.
To be honest, no vendor supports clusters without a fencing device so
we don't seriously test it.

>
>>
>> Basically at the point a resource fails and stonith is not configured
>> - shutdown is best-effort.
>
> I agree, shutdown of the failed resource can be best-effort ..
> especially on cluster shutdown ... even then I'd like to have a choice.
>
> I don't agree that ignoring the stop order requirements is also
> best-effort on "simple" resource stop/migration ... I'd like to tweak my
> resource to insist on stop order for dependent resources.

The problem (normally) is that the cluster is being given conflicting
instructions.
It can't both shut down and observe the constraint.

Probably the cluster should not have tried to shut down the node as a
substitute for lack of stonith.