<HTML>
<HEAD>
<TITLE>How to prevent locked I/O using Pacemaker with Primary/Primary DRBD/OCFS2 (Ubuntu 10.10)</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>All,<BR>
<BR>
I am running a two-node web cluster on OCFS2 (v1.5.0) via DRBD Primary/Primary (v8.3.8) and Pacemaker. Everything seems to be working great, except during testing of hard-boot scenarios.<BR>
<BR>
Whenever I hard-boot one of the nodes, the other node is successfully fenced and marked “Outdated”<BR>
<BR>
* <resource minor="0" cs="WFConnection" ro1="Primary" ro2="Unknown"ds1="UpToDate" ds2="Outdated" /><BR>
<BR>
However, this locks up I/O on the still active node and prevents any operations within the cluster :( I have even forced DRBD into StandAlone mode while in this state, but that does not resolve the I/O lock either....does anyone know if this is possible using OCFS2 (maintaining an active cluster in Primary/Unknown once the other node has a failure? E.g. Be it forced, controlled, etc)<BR>
<BR>
I have been focusing on DRBD config, but I am starting to wonder if perhaps it’s something with my Pacemaker or OCFS2 setup that is forcing this I/O lock during a failure. Any thoughts?<BR>
<B><BR>
</B>-----------------------------<BR>
<B>crm_mon (crm_mon 1.0.9 for OpenAIS and Heartbeat):<BR>
</B><BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>============<BR>
Last updated: Mon Apr 4 12:57:47 2011<BR>
Stack: openais<BR>
Current DC: ubu10a - partition with quorum<BR>
Version: 1.0.9-unknown<BR>
2 Nodes configured, 2 expected votes<BR>
4 Resources configured.<BR>
============<BR>
<BR>
Online: [ ubu10a ubu10b ]<BR>
<BR>
Master/Slave Set: msDRBD<BR>
Masters: [ ubu10a ubu10b ]<BR>
Clone Set: cloneDLM<BR>
Started: [ ubu10a ubu10b ]<BR>
Clone Set: cloneO2CB<BR>
Started: [ ubu10a ubu10b ]<BR>
Clone Set: cloneFS<BR>
Started: [ ubu10a ubu10b ]<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
-----------------------------<BR>
<B>DRBD (v8.3.8):<BR>
</B></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
version: 8.3.8 (api:88/proto:86-94)<BR>
0:repdata Connected Primary/Primary UpToDate/UpToDate C /data ocfs2<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
-----------------------------<BR>
<B>DRBD Conf:<BR>
</B></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
global {<BR>
usage-count no;<BR>
}<BR>
common {<BR>
syncer { rate 10M; }<BR>
}<BR>
resource repdata {<BR>
protocol C;<BR>
<BR>
meta-disk internal;<BR>
device /dev/drbd0;<BR>
disk /dev/sda3;<BR>
<BR>
handlers {<BR>
pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";<BR>
pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";<BR>
local-io-error "echo o > /proc/sysrq-trigger ; halt -f";<BR>
split-brain "/usr/lib/drbd/notify-split-brain.sh root";<BR>
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";<BR>
after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";<BR>
}<BR>
startup {<BR>
degr-wfc-timeout 120; # 120 = 2 minutes.<BR>
wfc-timeout 30;<BR>
become-primary-on both;<BR>
}<BR>
disk {<BR>
fencing resource-only;<BR>
}<BR>
syncer {<BR>
rate 10M;<BR>
al-extents 257;<BR>
}<BR>
net {<BR>
cram-hmac-alg "sha1";<BR>
shared-secret "XXXXXXX";<BR>
allow-two-primaries;<BR>
after-sb-0pri discard-zero-changes;<BR>
after-sb-1pri discard-secondary;<BR>
after-sb-2pri disconnect;<BR>
}<BR>
on ubu10a {<BR>
address 192.168.0.66:7788;<BR>
}<BR>
on ubu10b {<BR>
address 192.168.0.67:7788;<BR>
}<BR>
}<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
-----------------------------<BR>
<B>CIB.xml<BR>
</B></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
node ubu10a \<BR>
attributes standby="off"<BR>
node ubu10b \<BR>
attributes standby="off"<BR>
primitive resDLM ocf:pacemaker:controld \<BR>
op monitor interval="120s"<BR>
primitive resDRBD ocf:linbit:drbd \<BR>
params drbd_resource="repdata" \<BR>
operations $id="resDRBD-operations" \<BR>
op monitor interval="20s" role="Master" timeout="120s" \<BR>
op monitor interval="30s" role="Slave" timeout="120s"<BR>
primitive resFS ocf:heartbeat:Filesystem \<BR>
params device="/dev/drbd/by-res/repdata" directory="/data" fstype="ocfs2" \<BR>
op monitor interval="120s"<BR>
primitive resO2CB ocf:pacemaker:o2cb \<BR>
op monitor interval="120s"<BR>
ms msDRBD resDRBD \<BR>
meta resource-stickines="100" notify="true" master-max="2" interleave="true"<BR>
clone cloneDLM resDLM \<BR>
meta globally-unique="false" interleave="true"<BR>
clone cloneFS resFS \<BR>
meta interleave="true" ordered="true"<BR>
clone cloneO2CB resO2CB \<BR>
meta globally-unique="false" interleave="true"<BR>
colocation colDLMDRBD inf: cloneDLM msDRBD:Master<BR>
colocation colFSO2CB inf: cloneFS cloneO2CB<BR>
colocation colO2CBDLM inf: cloneO2CB cloneDLM<BR>
order ordDLMO2CB 0: cloneDLM cloneO2CB<BR>
order ordDRBDDLM 0: msDRBD:promote cloneDLM<BR>
order ordO2CBFS 0: cloneO2CB cloneFS<BR>
property $id="cib-bootstrap-options" \<BR>
dc-version="1.0.9-unknown" \<BR>
cluster-infrastructure="openais" \<BR>
stonith-enabled="false" \<BR>
no-quorum-policy="ignore" \<BR>
expected-quorum-votes="2"<BR>
<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><B><BR>
</B>----------------------------- <BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</SPAN></FONT>
</BODY>
</HTML>