[Pacemaker] Pacemaker on system with disk failure

[John Lauro]

Had this problem with heartbeat too (couldn't even failover/takeover to good node because the other node was stuck in a particular state from the read only filesystem), and so I have my own system health checks that in this case will reboot the system if it detects a file system not behaving. Basically it runs:
echo b > /proc/sysrq-trigger
(also had this problem with non clustered filesystems.)


It's not that frequent, but for example a network hick-up on a SAN can cause it, especially if you leave the default timeouts...

(Probably don't have to use ksh, I just always do...)

Example filesystem-watchdog:
#! /bin/ksh

fst=/filesystem-test
st=/proc/sysrq-trigger

while true ; do
  /bin/touch ${fst} || echo b > $st
  [ ! -f ${fst} ] && echo b > $st
  /bin/rm -f ${fst} || echo b > $st
  [ -f ${fst} ] && echo b > $st
  sleep 20
done


(Just run the script in the background as a type of watch dog that will auto-reboot in case of problem...)

(My actual implementation is a bit more complicated, but the main points is to keep it simple enough that the entire script is kept in cache and that it doesn't have to read anything different such as /usr/sbin/init when the filesystem is broken).  This method is fine either as a background process with that while loop, or you could just do once a minute from cron, etc...


Once the node kills itself, all should be well as it will no longer be stuck in a state where it can't update it's status...

That said, pacemaker should figure out that it's filesystem is broke and shut itself down or something...

It's a bit of a kludge, but... it works... (haven't tested with pacemaker, but I assume it should help your case...)
Hope you find it useful.



----- Original Message -----
> From: "Carsten Otto" <carsten.otto at andrena.de>
> To: "Digimer" <lists at alteeve.ca>
> Cc: "The Pacemaker cluster resource manager" <pacemaker at oss.clusterlabs.org>
> Sent: Thursday, September 25, 2014 8:28:08 AM
> Subject: Re: [Pacemaker] Pacemaker on system with disk failure
> 
> On Tue, Sep 23, 2014 at 10:14:33AM -0400, Digimer wrote:
> > You don't have real fencing configured, by the looks of it. Without
> > real, working fencing, recovery can be unpredictable. Can you set
> > that up and see if the problem goes away?
> 
> I now have real fencing which also is fully automatic, non-manual.
> The
> problem still exists just as described in previous mails.
> 
> The only error seen by the surviving node is that the stonith
> resource
> of the diskless node failed. However, this does not cause a
> switchover.
> 
> As said earlier, the services are not monitored anymore, and they do
> not
> work. Yet, the resources stay on the diskless node, as if nothing
> happened.
> 
> As far as I understand the situation, this is a grave error. A mere
> disk
> failure causes the whole setup to be in a failed state. There is no
> working monitoring, no switchover happens, ... From a client's
> perspective there's no difference to a non-redundant setup.
> 
> If you find the time, please try it out yourself. Just pull the
> cable from the/all disks which provide the root filesystem.
> 
> Best regards,
> Carsten
> --
> andrena objects ag
> Büro Frankfurt
> Clemensstr. 8
> 60487 Frankfurt
> 
> Tel: +49 (0) 69 977 860 38
> Fax: +49 (0) 69 977 860 39
> http://www.andrena.de
> 
> Vorstand: Hagen Buchwald, Matthias Grund, Dr. Dieter Kuhn
> Aufsichtsratsvorsitzender: Rolf Hetzelberger
> 
> Sitz der Gesellschaft: Karlsruhe
> Amtsgericht Mannheim, HRB 109694
> USt-IdNr. DE174314824
> 
> Bitte beachten Sie auch unsere anstehenden Veranstaltungen:
> http://www.andrena.de/events
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>