[Pacemaker] Pacemaker on system with disk failure

[Digimer]

You don't have real fencing configured, by the looks of it. Without 
real, working fencing, recovery can be unpredictable. Can you set that 
up and see if the problem goes away?

digimer

On 23/09/14 09:59 AM, Carsten Otto wrote:
> On Tue, Sep 23, 2014 at 09:50:12AM -0400, Digimer wrote:
>> Can you share your pacemaker and drbd configurations please?
>
> drbd.d/global_comman.conf:
> global {
>    usage-count no;
> }
>
> common {
>    protocol C;
>    handlers {
>      split-brain "/usr/lib/drbd/notify-split-brain.sh root";
>      out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
>    }
> }
>
> drbd.d/disk0.res:
> resource disk0 {
>          syncer {
>                  rate 10M;
>                  csums-alg sha1;
>          }
>          disk {
>                  on-io-error detach;
>                  fencing resource-only;
>          }
>          handlers {
>                  before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh";
>                  after-resync-target "/usr/lib/drbd/unsnapshot-resync-target-lvm.sh; /usr/lib/drbd/crm-unfence-peer.sh";
>                  fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
>                  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
>          }
>          net {
>                  after-sb-0pri discard-younger-primary;
>                  after-sb-1pri discard-secondary;
>                  after-sb-2pri call-pri-lost-after-sb;
>          }
>          device    /dev/drbd0;
>          disk      /dev/centos/drbd-lv;
>          meta-disk internal;
>          on node_a {
>                  address   192.168.69.89:7789;
>          }
>          on node_b {
>                  address   192.168.69.90:7789;
>
>          }
> }
>
> pcs resource --full:
>   Master: DRBD_MASTER
>    Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true failure-timeout=60sec
>    Resource: DRBD (class=ocf provider=linbit type=drbd)
>     Attributes: drbd_resource=disk0
>     Meta Attrs: failure-timeout=60sec
>     Operations: start interval=0s timeout=240 (DRBD-start-timeout-240)
>                 promote interval=0s timeout=90 (DRBD-promote-timeout-90)
>                 demote interval=0s timeout=90 (DRBD-demote-timeout-90)
>                 stop interval=0s timeout=100 (DRBD-stop-timeout-100)
>                 monitor interval=9 role=Master (DRBD-monitor-interval-9)
>                 monitor interval=11 role=Slave (DRBD-monitor-interval-11)
>   Group: GROUP
>    Resource: VIP (class=ocf provider=heartbeat type=IPaddr2)
>     Attributes: ip=192.168.69.48 cidr_netmask=32
>     Meta Attrs: failure-timeout=60sec
>     Operations: start interval=0s timeout=20s (VIP-start-timeout-20s)
>                 stop interval=0s timeout=5s (VIP-stop-timeout-5s)
>                 monitor interval=10sec (VIP-monitor-interval-10sec)
>    Resource: FS (class=ocf provider=heartbeat type=Filesystem)
>     Attributes: device=/dev/drbd0 directory=/mnt/drbd options=noatime,nodiratime fstype=ext4
>     Meta Attrs: failure-timeout=60sec
>     Operations: start interval=0s timeout=60 (FS-start-timeout-60)
>                 stop interval=0s timeout=10s (FS-stop-timeout-10s)
>                 monitor interval=5sec (FS-monitor-interval-5sec)
>    Resource: PGSQL (class=ocf provider=heartbeat type=pgsql)
>     Meta Attrs: failure-timeout=60sec
>     Operations: start interval=0s timeout=120 (PGSQL-start-timeout-120)
>                 stop interval=0s timeout=120 (PGSQL-stop-timeout-120)
>                 promote interval=0s timeout=120 (PGSQL-promote-timeout-120)
>                 demote interval=0s timeout=120 (PGSQL-demote-timeout-120)
>                 monitor interval=10sec (PGSQL-monitor-interval-10sec)
>    Resource: ASTERISK (class=ocf provider=heartbeat type=asterisk)
>     Meta Attrs: failure-timeout=60sec
>     Operations: start interval=0s timeout=20 (ASTERISK-start-timeout-20)
>                 monitor interval=10sec (ASTERISK-monitor-interval-10sec)
>                 stop interval=0s timeout=1 (ASTERISK-stop-timeout-1)
>    Resource: TOMCAT (class=ocf provider=heartbeat type=tomcat)
>     Attributes: java_home=/usr/java/latest/ catalina_home=/usr/share/tomcat statusurl=http://localhost:8080/xxx/
>     Meta Attrs: failure-timeout=60sec
>     Operations: start interval=0s timeout=60s (TOMCAT-start-timeout-60s)
>                 stop interval=0s timeout=20s (TOMCAT-stop-timeout-20s)
>                 monitor interval=10sec (TOMCAT-monitor-interval-10sec)
>
> pcs constraint --full:
> Location Constraints:
>    Resource: DRBD_MASTER
>      Constraint: drbd-fence-by-handler-disk0-DRBD_MASTER
>        Rule: score=-INFINITY role=Master  (id:drbd-fence-by-handler-disk0-rule-DRBD_MASTER)
>          Expression: #uname ne node_a  (id:drbd-fence-by-handler-disk0-expr-DRBD_MASTER)
>    Resource: STONITH_A
>      Disabled on: node_b (score:-INFINITY) (id:location-STONITH_A-node_b--INFINITY)
> Ordering Constraints:
>    promote DRBD_MASTER then start GROUP (Mandatory) (id:order-DRBD_MASTER-GROUP-mandatory)
> Colocation Constraints:
>    GROUP with DRBD_MASTER (INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-GROUP-DRBD_MASTER-INFINITY)
>
> pcs stonith --full:
>   STONITH_A  (stonith:fence_dummy):  Started
>   Resource: STONITH_A (class=stonith type=fence_dummy)
>    Attributes: passwd=x pcmk_host_list=node_b
>    Operations: monitor interval=60s (STONITH_A-monitor-interval-60s)
>
> [Note: The problem also happens without stonith and with a proper stonith
> configuration on both nodes!]
>
> pcs property:
> Cluster Properties:
>   cluster-infrastructure: corosync
>   cluster-recheck-interval: 5min
>   dc-version: 1.1.10-32.el7_0-368c726
>   last-lrm-refresh: 1411475550
>   no-quorum-policy: ignore
>   stonith-enabled: true
>


-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?