<div dir="ltr">Thanks.<div><br></div><div>I missed</div><div><br></div><div>drbdadm -- --force create-md vlv<br></div><div><br></div><div>on node2, that was the cause of the problem. Because of this there was no drbd-minor-0.lkbd file in /var/lib/drbd on node2, so it did not work properly (e.g. "drbdadm adjust" did not work, and that killed pacemaker).</div><div><br></div><div>BTW to speedup initial resync, one could execute:</div><div><br></div><div>drbdadm -- --force create-md vlv<br></div><div>drbdadm up vlv<br></div><div><div>drbdadm -- --clear-bitmap new-current-uuid vlv/0</div><div>drbdadm primary vlv</div><div>mkfs.xfs -d agcount=8 /dev/drbd0<br></div></div><div><br></div><div><br></div><div>P.S.<br>I played with strace and sources a bit and discovered that "service drbd start" looked working only because it executes "drbdadm adjust-with-progress", although pacemaker's script executes "drbdadm -v adjust". In contrast to what is said in the docs, adjust and adjust-with-progress differ not only by debug output: there used to be a hack in drbdadm code which forces them to work differently:</div><div><br></div><div><div> for (stage = CFG_PREREQ; stage < __CFG_LAST; stage++) {</div><div> r = _run_deferred_cmds(stage);</div><div> if (r) {</div><div> if (!adjust_with_progress)</div><div> return 1; /* FIXME r? */</div><div> ret = 1;</div><div> }</div><div> }</div></div><div><br></div><div>This "FIXME" code is executed in "adjust" mode, but NOT when "adjust-with-progress" is active. So if one uses "adjust-with-progress", drbdadm silently skips failed steps and continues with the next ones, while "adjust" fails on a first failed step.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 3, 2015 at 10:27 PM, Vladislav Bogdanov <span dir="ltr"><<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">03.01.2015 20:35, Dmitry Koterov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello.<br>
<br>
Ubuntu 14.04, corosync 2.3.3, pacemaker 1.1.10. The cluster consists of<br>
2 nodes (node1 and node2), when I run "crm node standby node2" and then,<br>
in a minute, "crm node online node2", DRBD secondary on node2 does not<br>
start. Logs say that "drbdadm -c /etc/drbd.conf check-resize vlv" fails<br>
with an error message: "No valid meta data found" on the onlining node.<br>
And, surprisingly, after I run "service drbd start" on node2 manually,<br>
everything becomes fine.<br>
<br>
Maybe something is broken in /usr/lib/ocf/resource.d/<u></u>linbit/drbd, why<br>
cannot it start DRBD? Or I am misconfigured somehow? Could you please<br>
give an advice what to do?<br>
</blockquote>
<br></span>
Please see inline in drbdadm output.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
I have the following configuration (drbd + mount + postgresql, but<br>
postgresql is innocent here, so just ignore it):<br>
<br></span>
*root@node2:/var/log#* crm configure show<div><div class="h5"><br>
node $id="1017525950" node2 attributes standby="off"<br>
node $id="1760315215" node1<br>
primitive drbd ocf:linbit:drbd \<br>
params drbd_resource="vlv" \<br>
op start interval="0" timeout="240" \<br>
op stop interval="0" timeout="120"<br>
primitive fs ocf:heartbeat:Filesystem \<br>
params device="/dev/drbd0" directory="/var/lib/vlv.drbd/<u></u>root"<br>
options="noatime,nodiratime" fstype="xfs" \<br>
op start interval="0" timeout="300" \<br>
op stop interval="0" timeout="300"<br>
primitive postgresql lsb:postgresql \<br>
op monitor interval="4" timeout="60" \<br>
op start interval="0" timeout="60" \<br>
op stop interval="0" timeout="60"<br>
group pgserver fs postgresql<br>
ms ms_drbd drbd \<br>
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"<br>
notify="true"<br>
location cli-prefer-pgserver pgserver inf: node1<br>
colocation col_pgserver inf: pgserver ms_drbd:Master<br>
order ord_pgserver inf: ms_drbd:promote pgserver:start<br>
property $id="cib-bootstrap-options" dc-version="1.1.10-42f2063"<br>
cluster-infrastructure="<u></u>corosync"<br>
stonith-enabled="false" no-quorum-policy="ignore"<br>
last-lrm-refresh="1420304078"<br>
rsc_defaults $id="rsc-options" \<br>
resource-stickiness="100"<br>
<br>
<br>
The cluster and DRBD statuses on node2 look healthy:<br>
<br>
<br></div></div>
*root@node2:/var/log#* crm status<span class=""><br>
...<br>
Online: [ node1 node2 ]<br>
Master/Slave Set: ms_drbd [drbd]<br>
Masters: [ node1 ]<br>
Slaves: [ node2 ]<br>
Resource Group: pgserver<br>
fs(ocf::heartbeat:Filesystem):<u></u>Started node1<br>
postgresql(lsb:postgresql):<u></u>Started node1<br>
<br></span>
*root@node2:/var/log#* cat /proc/drbd<span class=""><br>
version: 8.4.3 (api:1/proto:86-101)<br>
srcversion: F97798065516C94BE0F27DC<br>
0: cs:Connected ro:Secondary/Primary ds:Diskless/UpToDate C r-----<br>
</span></blockquote>
^^^^^^^^<br>
that is your problem<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
<br>
<br>
Now I switch node2 to standby and verify that DRBD on it has really shot<br>
down:<br>
<br>
<br></span>
*root@node1:/etc/rc2.d#* crm node standby node2<br>
*root@node2:/var/log#* cat /proc/drbd<span class=""><br>
version: 8.4.3 (api:1/proto:86-101)<br>
srcversion: F97798065516C94BE0F27DC<br></span>
*root@node2:/var/log#*<br>
*<span class=""><br>
*<br>
Then I switch node2 back online and see that DRBD has not been<br>
initialized and reattached again!<br>
<br></span>
*root@node2:/var/log#* >syslog<br>
*root@node1:/etc#* crm node online node2<br>
*root@node2:/var/log#* crm status<span class=""><br>
...<br>
Online: [ node1 node2 ]<br>
Master/Slave Set: ms_drbd [drbd]<br>
Masters: [ node1 ]<br>
Stopped: [ node2 ]<br>
Resource Group: pgserver<br>
fs(ocf::heartbeat:Filesystem):<u></u>Started node1<br></span>
postgresql(lsb:postgresql):<u></u>Started node1<span class=""><br>
Failed actions:<br>
drbd_start_0 (node=node2, call=81, rc=1, status=complete,<br>
last-rc-change=Sat Jan 3 12:05:32 2015<br>
, queued=1118ms, exec=0ms<br>
): unknown error<br>
<br></span>
*root@node2:/var/log#* cat syslog | head -n 30<div><div class="h5"><br>
Jan 3 12:05:31 node2 crmd[918]: notice: do_state_transition: State<br>
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC<br>
cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>
Jan 3 12:05:31 node2 cib[913]: notice: cib:diff: Diff: --- 0.29.3<br>
Jan 3 12:05:31 node2 cib[913]: notice: cib:diff: Diff: +++ 0.30.1<br>
027344551b46745123e4a52562e559<u></u>74<br>
Jan 3 12:05:31 node2 pengine[917]: notice: unpack_config: On loss of<br>
CCM Quorum: Ignore<br>
Jan 3 12:05:31 node2 pengine[917]: notice: LogActions: Start<br>
drbd:1#011(node2)<br>
Jan 3 12:05:31 node2 crmd[918]: notice: te_rsc_command: Initiating<br>
action 46: notify drbd_pre_notify_start_0 on node1<br>
Jan 3 12:05:31 node2 pengine[917]: notice: process_pe_message:<br>
Calculated Transition 11: /var/lib/pacemaker/pengine/pe-<u></u>input-11.bz2<br>
Jan 3 12:05:32 node2 crmd[918]: notice: te_rsc_command: Initiating<br>
action 10: start drbd:1_start_0 on node2 (local)<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf check-resize vlv<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Exit code 255<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Command output:<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf --peer node1 attach vlv<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Exit code 255<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Command output:<br>
Jan 3 12:05:33 node2 drbd(drbd)[1931]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf --peer node1 attach vlv<br>
Jan 3 12:05:33 node2 drbd(drbd)[1931]: ERROR: vlv: Exit code 255<br>
Jan 3 12:05:33 node2 drbd(drbd)[1931]: ERROR: vlv: Command output:<br>
Jan 3 12:05:33 node2 lrmd[915]: notice: operation_finished:<br>
drbd_start_0:1931:stderr [ no suitable meta data found :( ]<br>
Jan 3 12:05:33 node2 lrmd[915]: notice: operation_finished:<br>
drbd_start_0:1931:stderr [ Command 'drbdmeta 0 v08 /dev/loop0 internal<br>
check-resize' terminated with exit code 255 ]<br>
Jan 3 12:05:33 node2 lrmd[915]: notice: operation_finished:<br>
drbd_start_0:1931:stderr [ No valid meta data found ]<br>
Jan 3 12:05:33 node2 lrmd[915]: notice: operation_finished:<br>
drbd_start_0:1931:stderr [ Command 'drbdmeta 0 v08 /dev/loop0 internal<br>
apply-al' terminated with exit code 255 ]<br>
Jan 3 12:05:33 node2 lrmd[915]: notice: operation_finished:<br>
drbd_start_0:1931:stderr [ No valid meta data found ]<br>
Jan 3 12:05:33 node2 lrmd[915]: notice: operation_finished:<br>
drbd_start_0:1931:stderr [ Command 'drbdmeta 0 v08 /dev/loop0 internal<br>
apply-al' terminated with exit code 255 ]<br>
Jan 3 12:05:33 node2 crmd[918]: notice: process_lrm_event: LRM<br>
operation drbd_start_0 (call=81, rc=1, cib-update=79, confirmed=true)<br>
unknown error<br>
Jan 3 12:05:33 node2 crmd[918]: notice: process_lrm_event:<br>
node2-drbd_start_0:81 [ \n\n\n\n\n\n\n ]<br>
Jan 3 12:05:33 node2 crmd[918]: warning: status_from_rc: Action 10<br>
(drbd:1_start_0) on node2 failed (target: 0 vs. rc: 1): Error<br>
Jan 3 12:05:33 node2 crmd[918]: warning: update_failcount: Updating<br>
failcount for drbd on node2 after failed start: rc=1 (update=INFINITY,<br>
time=1420304733)<br>
Jan 3 12:05:33 node2 attrd[916]: notice: attrd_trigger_update:<br>
Sending flush op to all hosts for: fail-count-drbd (INFINITY)<br>
Jan 3 12:05:33 node2 attrd[916]: notice: attrd_perform_update: Sent<br>
update 60: fail-count-drbd=INFINITY<br>
Jan 3 12:05:33 node2 attrd[916]: notice: attrd_trigger_update:<br>
Sending flush op to all hosts for: last-failure-drbd (1420304733)<br>
<br>
root@node2:/var/log# cat syslog | grep ERROR | head -n 30<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf check-resize vlv<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Exit code 255<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Command output:<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf --peer node1 attach vlv<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Exit code 255<br>
Jan 3 12:05:32 node2 drbd(drbd)[1931]: ERROR: vlv: Command output:<br>
Jan 3 12:05:33 node2 drbd(drbd)[1931]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf --peer node1 attach vlv<br>
Jan 3 12:05:33 node2 drbd(drbd)[1931]: ERROR: vlv: Exit code 255<br>
Jan 3 12:05:33 node2 drbd(drbd)[1931]: ERROR: vlv: Command output:<br>
Jan 3 12:05:33 node2 drbd(drbd)[2021]: ERROR: vlv: Called drbdadm -c<br>
/etc/drbd.conf --peer node1 -v adjust vlv<br>
Jan 3 12:05:33 node2 drbd(drbd)[2021]: ERROR: vlv: Exit code 1<br>
Jan 3 12:05:33 node2 drbd(drbd)[2021]: ERROR: vlv: Command output:<br>
drbdmeta 0 v08 /dev/loop0 internal apply-al<br>
<br>
<br>
Here's the DRBD config file:<br>
<br>
<br></div></div>
*root@node2:/var/log#* cat /etc/drbd.d/drbd.res<span class=""><br>
resource vlv {<br>
device /dev/drbd0;<br>
disk /dev/loop0;<br>
meta-disk internal;<br>
syncer { rate 150M; verify-alg md5; }<br>
on node1 { address x.x.x.x:7788; }<br>
on node2 { address y.y.y.y:7788; }<br>
}<br>
<br></span>
*root@node2:/var/log#* losetup -a<br>
/dev/loop0: [fd01]:1314858 (/var/lib/vlv.drbd/vlv.img)<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br></div>