[Pacemaker] Manual fsck required on passive node upon failover

[Daniel Qian]

----- Original Message ----- 
From: "Raoul Bhatia [IPAX]" <r.bhatia at ipax.at>
To: <pacemaker at oss.clusterlabs.org>; "Daniel Qian" <daniel at bestningning.com>
Sent: Thursday, December 10, 2009 11:10 AM
Subject: Re: [Pacemaker] Manual fsck required on passive node upon failover


> On 12/10/2009 05:01 PM, Daniel Qian wrote:
>> I installed a two-node cluster following this link on clusterlabs.org
>> http://www.howtoforge.com/installation-and-setup-guide-for-drbd-openais-pacemaker-xen-on-opensuse-11.1
>>
>> The guide is for OpenSuse but I did it on Centos 5 as all the packages
>> are available there.
>>
>> Basically what I did so far is a Xen guest instance sitting on the
>> DRBD-backed file system. Everything seems to be working fine except for
>> one thing - When I shutdown openais (service openais stop) on the active
>> node for failover testing, the passive node was trying to take over all
>> the resources but got stuck at the point of file system resource. Logs
>> show it required file system check thus manual intervention is required
>> to bring it online. However if I do 'crm resource move xen_rsc
>> passive_node' the failover transits fine every time.
>
> can you reproduce the problem or has this been a one time issue?

This is not a one time issue. it happened more than half of the times I did 
the test. My hunch is the Xen instance did not get shutdown cleanly in time 
or the file system did not get umounted on the active node. I ended up with 
primary/primary for drbd roles of the two nodes when that happened.

> did you turn off the "periodic fsck" on e.g. ext3? see:
>
> # tune2fs -l /dev/md0
> ...
> Filesystem state:         clean
> Errors behavior:          Continue
> ..
> Mount count:              2
> Maximum mount count:      23
> Last checked:             Sat Aug  8 16:17:11 2009
> Check interval:           15552000 (6 months)
> Next check after:         Thu Feb  4 15:17:11 2010
> ..
>

This doesnt seem to be the issue

dev/drbd0           691435652   1945052 654367656   1% /xen
[root at ilo142 ~]# tune2fs -l /dev/drbd0
tune2fs 1.39 (29-May-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          94351cb7-d51d-4d63-8c31-e261e12f8647
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index 
filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              87818240
Block count:              175614727
Reserved block count:     8780736
Free blocks:              172388127
Free inodes:              87818227
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      982
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512


Thanks,
Daniel