[Pacemaker] Cluster Volume Group is stuck

[Lars Marowsky-Bree]

On 2011-05-12T15:16:52, Karl Rößmann <K.Roessmann at fkf.mpg.de> wrote:

> This is an Update to my last Mail:
> 
> SBD is running on one Node normally:

I didn't mean to inquire wrt the external/sbd fencing agent, but the
system daemon "sbd" - as configured via /etc/sysconfig/sbd and started
(automatically) via /etc/init.d/openais (on SLE HA).

> but after powering off Node multix246, it is running on two nodes:
> 
> 
> Node multix246: UNCLEAN (offline)
> Online: [ multix244 multix245 ]
> 
>  Clone Set: dlm_clone [dlm]
>      Started: [ multix244 multix245 ]
>      Stopped: [ dlm:2 ]
>  Clone Set: clvm_clone [clvm]
>      Started: [ multix244 multix245 ]
>      Stopped: [ clvm:2 ]
>  Clone Set: vgsmet_clone [vgsmet]
>      Started: [ multix244 multix245 ]
>      Stopped: [ vgsmet:2 ]
>  smetserv       (ocf::heartbeat:Xen):   Started multix244
>  SBD_Stonith    (stonith:external/sbd) Started [ multix245  multix246 ] <-----

That is normal. It was running on the x246 node previously, but to fence
said node, it needs to be started in the local partition.

Normally, some seconds later, the fence should complete and multix246
should change state to "OFFLINE". The state you see above is only
transient.

If it remains stuck in this state for more longer, I would assume the
fence targetting multix246 isn't actually completing; do you see the
fence/stonith request being issued in the logs? Is there an error
message from sbd on multix245 in the above scenario?

Ah! Got it -

>From your other mail, seeing that 

> sbd -d /dev/disk/by-id/scsi-3600a0b8000420d5a00001cf14dc3a9a2-part1 list
> 0       multix244       clear
> 1       multix245       clear
> 2       multix246       reset   multix245

suggests that multix246 actually was sent the request; and thus, should
be considered 'fenced' by the remaining cluster.

Looking back in your mails further:

>> /dev/disk/by-id/scsi-3600a0b8000420d5a00001cf14dc3a9a2-part1 dump
>> Header version     : 2
>> Number of slots    : 255
>> Sector size        : 512
>> Timeout (watchdog) : 60
>> Timeout (allocate) : 2
>> Timeout (loop)     : 1
>> Timeout (msgwait)  : 120

You've set extremely long timeouts for the watchdog, and in particular
for the msgwait - this means that a fence will only be considered
completed after 120s by sbd. At the same time, you've set
stonith-timeout to 60s, so if the fence takes longer than that, it'll be
considered failed.

You've set up your cluster so that it can never complete a successful
fence - congratulations! ;-)

If you've got a legitimate reason for setting the msgwait timeout to
120s, you need to set the stonith-timeout to >120s - 140s, for example.


Regards,
    Lars

-- 
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde