[Pacemaker] failed over filesystem mount points not coming up on secondary node
Lonni J Friedman
netllama at gmail.com
Mon Oct 1 16:31:05 EDT 2012
I'm still dead in the water here, and could really use some clues.
I tried tweaking my config a bit to simplify it, in the hope that it
would at least work with fewer resources, but that too fails in the
exact same fashion. Specifically, the DRBD resource does failover to
promote the old slave to a master, but the failover IP never gets
promoted, and the DRBD backed block device is never mounted on the new
master.
farm-ljf1 used to be the master for all resources. I stopped
corosync, intending to failover everything to farm-ljf0. Since I did
that, here's how things look:
##########
[root at farm-ljf0 ~]# crm status
============
Last updated: Mon Oct 1 13:06:07 2012
Last change: Mon Oct 1 12:17:16 2012 via cibadmin on farm-ljf1
Stack: openais
Current DC: farm-ljf0 - partition WITHOUT quorum
Version: 1.1.7-2.fc16-ee0730e13d124c3d58f00016c3376a1de5323cff
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ farm-ljf0 ]
OFFLINE: [ farm-ljf1 ]
Master/Slave Set: FS0_Clone [FS0]
Masters: [ farm-ljf0 ]
Stopped: [ FS0:1 ]
Failed actions:
FS0_drbd_start_0 (node=farm-ljf0, call=53, rc=1, status=complete):
unknown error
##########
I looked in /var/log/cluster/corosync.log from the time when I
attempted the failover, and spotted the following:
#########
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: rsc:FS0_drbd:53: start
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output:
(FS0_drbd:start:stderr) blockdev:
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output:
(FS0_drbd:start:stderr) cannot open /dev/drbd0
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output:
(FS0_drbd:start:stderr) :
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output:
(FS0_drbd:start:stderr) Wrong medium type
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output:
(FS0_drbd:start:stderr) mount: block device /dev/drbd0 is
write-protected, mounting read-only
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output: (FS0_drbd:start:stderr)
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output:
(FS0_drbd:start:stderr) mount: Wrong medium type
Oct 01 12:56:18 farm-ljf0 lrmd: [924]: info: RA output: (FS0_drbd:start:stderr)
Oct 01 12:56:18 farm-ljf0 crmd: [927]: info: process_lrm_event: LRM
operation FS0_drbd_start_0 (call=53, rc=1, cib-update=532,
confirmed=true) unknown error
Oct 01 12:56:18 farm-ljf0 crmd: [927]: WARN: status_from_rc: Action 40
(FS0_drbd_start_0) on farm-ljf0 failed (target: 0 vs. rc: 1): Error
Oct 01 12:56:18 farm-ljf0 crmd: [927]: WARN: update_failcount:
Updating failcount for FS0_drbd on farm-ljf0 after failed start: rc=1
(update=INFINITY, time=1349121378)
Oct 01 12:56:18 farm-ljf0 crmd: [927]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=FS0_drbd_last_failure_0, mag
ic=0:1;40:287:0:655c1af8-d2e8-4dfa-b084-4d4d36be8ade, cib=0.34.33) :
Event failed
#########
To my eyes, it looks like the attempt to mount the drbd backed storage
failed. I don't understand why, as I can manually mount it using the
exact same parameters in the configuration (which worked fine on the
master) after the failover. Perhaps there's some weird race condition
occurring where it tries to mount before the drbd server has failed
over?
None of that explains why the failover IP didn't come up on the (old)
slave. I don't see any errors or failures in the log with respect to
ClusterIP. All I see is:
#########
Oct 01 12:56:17 farm-ljf0 pengine: [926]: notice: LogActions: Move
ClusterIP (Started farm-ljf1 -> farm-ljf0)
Oct 01 12:56:17 farm-ljf0 crmd: [927]: info: te_rsc_command:
Initiating action 41: stop ClusterIP_stop_0 on farm-ljf1
#########
It looks like it never even tries to bring it up on the (old) slave.
Anyway, here's the configuration that I was using when all of the
above transpired:
##########
[root at farm-ljf0 ~]# crm configure show
node farm-ljf0 \
attributes standby="off"
node farm-ljf1
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="10.31.97.100" cidr_netmask="22" nic="eth1" \
op monitor interval="10s" \
meta target-role="Started"
primitive FS0 ocf:linbit:drbd \
params drbd_resource="r0" \
op monitor interval="10s" role="Master" \
op monitor interval="30s" role="Slave"
primitive FS0_drbd ocf:heartbeat:Filesystem \
params device="/dev/drbd0" directory="/mnt/sdb1" fstype="xfs"
group g_services FS0_drbd ClusterIP
ms FS0_Clone FS0 \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true"
location cli-prefer-ClusterIP ClusterIP \
rule $id="cli-prefer-rule-ClusterIP" inf: #uname eq farm-ljf1
colocation fs0_on_drbd inf: g_services FS0_Clone:Master
order FS0_drbd-after-FS0 inf: FS0_Clone:promote g_services
property $id="cib-bootstrap-options" \
dc-version="1.1.7-2.fc16-ee0730e13d124c3d58f00016c3376a1de5323cff" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
##########
On Thu, Sep 27, 2012 at 3:10 PM, Lonni J Friedman <netllama at gmail.com> wrote:
> Greetings,
> I've just started playing with pacemaker/corosync on a two node setup.
> At this point I'm just experimenting, and trying to get a good feel
> of how things work. Eventually I'd like to start using this in a
> production environment. I'm running Fedora16-x86_64 with
> pacemaker-1.1.7 & corosync-1.4.3. I have DRBD setup and working fine
> with two resources. I've verified that pacemaker is doing the right
> thing when initially configured. Specifically:
> * the floating static IP is brought up
> * DRBD is brought up correctly with a master & slave
> * the local DRBD backed mount points are mounted correctly
>
> Here's the configuration:
> #########
> node farm-ljf0 \
> attributes standby="off"
> node farm-ljf1
> primitive ClusterIP ocf:heartbeat:IPaddr2 \
> params ip="10.31.97.100" cidr_netmask="22" nic="eth1" \
> op monitor interval="10s"
> primitive FS0 ocf:linbit:drbd \
> params drbd_resource="r0" \
> op monitor interval="10" role="Master" \
> op monitor interval="30" role="Slave"
> primitive FS0_drbd ocf:heartbeat:Filesystem \
> params device="/dev/drbd0" directory="/mnt/sdb1" fstype="xfs"
> primitive FS1 ocf:linbit:drbd \
> params drbd_resource="r1" \
> op monitor interval="10s" role="Master" \
> op monitor interval="30s" role="Slave"
> primitive FS1_drbd ocf:heartbeat:Filesystem \
> params device="/dev/drbd1" directory="/mnt/sdb2" fstype="xfs"
> ms FS0_Clone FS0 \
> meta master-max="1" master-node-max="1" clone-max="2"
> clone-node-max="1" notify="true"
> ms FS1_Clone FS1 \
> meta master-max="1" master-node-max="1" clone-max="2"
> clone-node-max="1" notify="true"
> location cli-prefer-ClusterIP ClusterIP \
> rule $id="cli-prefer-rule-ClusterIP" inf: #uname eq farm-ljf1
> colocation fs0_on_drbd inf: FS0_drbd FS0_Clone:Master
> colocation fs1_on_drbd inf: FS1_drbd FS1_Clone:Master
> order FS0_drbd-after-FS0 inf: FS0_Clone:promote FS0_drbd
> order FS1_drbd-after-FS1 inf: FS1_Clone:promote FS1_drbd
> property $id="cib-bootstrap-options" \
> dc-version="1.1.7-2.fc16-ee0730e13d124c3d58f00016c3376a1de5323cff" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> stonith-enabled="false" \
> no-quorum-policy="ignore"
> #########
>
> However, when I attempted to simulate a failover situation (I shutdown
> the current master/primary node completely), not everything failed
> over correctly. Specifically, the mount points did not get mounted,
> even though the other two elements did failover correctly.
> 'farm-ljf1' is the node that I shutdown, farm-ljf0 is the node that I
> expected to inherit all of the resources. Here's the status:
> #########
> [root at farm-ljf0 ~]# crm status
> ============
> Last updated: Thu Sep 27 15:00:19 2012
> Last change: Thu Sep 27 13:59:42 2012 via cibadmin on farm-ljf1
> Stack: openais
> Current DC: farm-ljf0 - partition WITHOUT quorum
> Version: 1.1.7-2.fc16-ee0730e13d124c3d58f00016c3376a1de5323cff
> 2 Nodes configured, 2 expected votes
> 7 Resources configured.
> ============
>
> Online: [ farm-ljf0 ]
> OFFLINE: [ farm-ljf1 ]
>
> ClusterIP (ocf::heartbeat:IPaddr2): Started farm-ljf0
> Master/Slave Set: FS0_Clone [FS0]
> Masters: [ farm-ljf0 ]
> Stopped: [ FS0:0 ]
> Master/Slave Set: FS1_Clone [FS1]
> Masters: [ farm-ljf0 ]
> Stopped: [ FS1:0 ]
>
> Failed actions:
> FS1_drbd_start_0 (node=farm-ljf0, call=23, rc=1, status=complete):
> unknown error
> FS0_drbd_start_0 (node=farm-ljf0, call=24, rc=1, status=complete):
> unknown error
> #########
>
> I eventually brought up the shut down node (farm-ljf1) again, hoping
> that might at least bring things back into a good state, but its not
> working either, and is showing up as OFFLINE:
> ##########
> [root at farm-ljf1 ~]# crm status
> ============
> Last updated: Thu Sep 27 15:06:54 2012
> Last change: Thu Sep 27 14:49:06 2012 via cibadmin on farm-ljf1
> Stack: openais
> Current DC: NONE
> 2 Nodes configured, 2 expected votes
> 7 Resources configured.
> ============
>
> OFFLINE: [ farm-ljf0 farm-ljf1 ]
> ##########
>
>
> So at this point, I've got two problems:
> 0) FS mount failover isn't working. I'm hoping this is some silly
> configuration issue that can be easily resolved.
> 1) bringing the "failed" farm-ljf1 node back online doesn't seem to
> work automatically, and I can't figure out what kind of magic is
> needed.
>
>
> If this stuff is documented somewhere, I'll gladly read it, if someone
> can point me in the right direction.
>
> thanks!
More information about the Pacemaker
mailing list