[Pacemaker] node1 fencing itself after node2 being fenced

[Vladislav Bogdanov]

05.02.2014 20:10, Asgaroth wrote:
> On 05/02/2014 16:12, Digimer wrote:
>> You say it's working now? If so, excellent. If you have any troubles
>> though, please share your cluster.conf and 'pcs config show'.
>>
> 
> Hi Digimer, no its not working as I expect it to when I test a crash of
> node 2, clvmd goes in to a failed state and then node1 gets "shot in the
> head", other than that the config appears works fine with the minimal
> testing I have done so far :)
> 
> I have attached the cluster.conf and pcs config files to the email (with
> minimal obfuscation).

Hi,

I bet your problem comes from the LSB clvmd init script.
Here is what it does do:

===========
...
clustered_vgs() {
    ${lvm_vgdisplay} 2>/dev/null | \
        awk 'BEGIN {RS="VG Name"} {if (/Clustered/) print $1;}'
}

clustered_active_lvs() {
    for i in $(clustered_vgs); do
        ${lvm_lvdisplay} $i 2>/dev/null | \
        awk 'BEGIN {RS="LV Name"} {if (/[^N^O^T] available/) print $1;}'
    done
}

rh_status() {
    status $DAEMON
}
...
case "$1" in
...
  status)
    rh_status
    rtrn=$?
    if [ $rtrn = 0 ]; then
        cvgs="$(clustered_vgs)"
        echo Clustered Volume Groups: ${cvgs:-"(none)"}
        clvs="$(clustered_active_lvs)"
        echo Active clustered Logical Volumes: ${clvs:-"(none)"}
    fi
...
esac

exit $rtrn
=========

So, it not only looks for status of daemon itself, but also tries to
list volume groups. And this operation is blocked because fencing is
still in progress, and the whole cLVM thing (as well as DLM itself and
all other dependent services) is frozen. So your resource timeouts in
monitor operation, and then pacemaker asks it to stop (unless you have
on-fail=fence). Anyways, there is a big chance that stop will fail too,
and that leads again to fencing. cLVM is very fragile in my opinion
(although newer versions running on corosync2 stack seem to be much
better). And it is probably still doesn't work well when managed by
pacemaker in CMAN-based clusters, because it blocks globally if any node
in the whole cluster is online at the cman layer but doesn't run clvmd
(I checked last time with .99). And that was the same for all stacks,
until was fixed for corosync (only 2?) stack recently. The problem with
that is that you cannot just stop pacemaker on one node (f.e. for
maintenance), you should immediately stop cman as well (or run clvmd in
cman'ish way) - cLVM freezes on another node. This should be easily
fixable in clvmd code, but nobody cares.

Increasing timeout for LSB clvmd resource probably wont help you,
because blocked (because of DLM waits for fencing) LVM operations iirc
never finish.

You may want to search for clvmd OCF resource-agent, it is available for
SUSE I think. Although it is not perfect, it should work much better for
you.

Best,
Vladislav