I'm not sure that Filesystem resource causes this behaviour. I'm doing some tests now and taking logs.<div>I think it may be related to res-nfs group. Now I founded that portmap is started by upstart before pacemaker and may be it is the reason of fail.</div>
<div><br><div class="gmail_quote">2011/6/21 Dejan Muhamedagic <span dir="ltr"><<a href="mailto:dejanmm@fastmail.fm">dejanmm@fastmail.fm</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Vladislav,<br>
<div class="im"><br>
On Tue, Jun 21, 2011 at 05:38:21PM +0300, Vladislav Bogdanov wrote:<br>
> 21.06.2011 17:23, Dejan Muhamedagic wrote:<br>
> > On Tue, Jun 21, 2011 at 06:10:16PM +0400, Aleksander Malaev wrote:<br>
> >> How can I check this?<br>
> >> If I don't add this exportfs resource then cluster is become the fully<br>
> >> operational - all mounts are accesible and fail-over between nodes is<br>
> >> working as it should. May be I need to add some sort of delay between this<br>
> >> resources?<br>
> ><br>
> > If you need to do so (there's actually start-delay, but it<br>
> > should be deprecated), then some RA doesn't implement start<br>
> > action correctly. In this case, it looks like it's Filesystem,<br>
> > right? Since the filesystem is ocfs2 it may be that the cluster<br>
> > services supporting ocfs2 are not fast enough. At any rate,<br>
> > Filesystem shouldn't start before the filesystem is really<br>
> > mounted.<br>
><br>
> If I recall correctly from my totally failed experiments with ocfs2<br>
> (simultaneous kernel panic on all nodes running f13-x86_64 ;), this is<br>
> ocfs2-specific problem.<br>
><br>
> Although mount call returns success, ocfs2 filesystem may be not ready<br>
> for consumption for at least several seconds.<br>
<br>
</div>That sounds like a plausible explanation. Before trying to fix<br>
ocfs2, which may take time or be impossible, we can make<br>
Filesystem use monitor internally to exit only once the<br>
filesystem has really been mounted. But please somebody first<br>
open a bugzilla, this needs to be tracked.<br>
<br>
BTW, interestingly I cannot recall that anybody complained about<br>
this before. It obviously depends on the network, but still...<br>
<br>
Cheers,<br>
<font color="#888888"><br>
Dejan<br>
</font><div><div></div><div class="h5"><br>
> Best,<br>
> Vladislav<br>
><br>
> ><br>
> > If so, please file a bugzilla for it and attach hb_report of the<br>
> > incident.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Dejan<br>
> ><br>
> >> 2011/6/21 Dejan Muhamedagic <<a href="mailto:dejanmm@fastmail.fm">dejanmm@fastmail.fm</a>><br>
> >><br>
> >>> On Tue, Jun 21, 2011 at 05:56:40PM +0400, Aleksander Malaev wrote:<br>
> >>>> Sure, I'm using order constraint.<br>
> >>>> But it seems that it doesn't check monitor of the previous started<br>
> >>> resource.<br>
> >>><br>
> >>> It doesn't need to check monitor. The previous resource, if<br>
> >>> started, must be fully operational. If it's not, then the RA is<br>
> >>> broken.<br>
> >>><br>
> >>> Thanks,<br>
> >>><br>
> >>> Dejan<br>
> >>><br>
> >>>> 2011/6/21 Dejan Muhamedagic <<a href="mailto:dejanmm@fastmail.fm">dejanmm@fastmail.fm</a>><br>
> >>>><br>
> >>>>> Hi,<br>
> >>>>><br>
> >>>>> On Mon, Jun 20, 2011 at 11:40:04PM +0400, Александр Малаев wrote:<br>
> >>>>>> Hello,<br>
> >>>>>><br>
> >>>>>> I have configured pacemaker+ocfs2 cluster with shared storage<br>
> >>> connected<br>
> >>>>> by<br>
> >>>>>> FC.<br>
> >>>>>> Now I need to setup NFS export in Active/Active mode and I added all<br>
> >>>>> needed<br>
> >>>>>> resources and wrote the order of starting.<br>
> >>>>>> But then node is starting after reboot I got race condition between<br>
> >>>>>> Filesystem resource and exportfs.<br>
> >>>>>> Exportfs couldn't start because ocfs2 mountpoint isn't mounted yet.<br>
> >>>>>><br>
> >>>>>> How to tell ExportFS resource to start then filesystem resource will<br>
> >>> be<br>
> >>>>>> ready?<br>
> >>>>><br>
> >>>>> Use the order constraint? Or did I miss something? You already<br>
> >>>>> have some order constraints defined, so you should be able to<br>
> >>>>> manage.<br>
> >>>>><br>
> >>>>> Thanks,<br>
> >>>>><br>
> >>>>> Dejan<br>
> >>>>><br>
> >>>>>> crm config is the following:<br>
> >>>>>> node msk-nfs-gw01<br>
> >>>>>> node msk-nfs-gw02<br>
> >>>>>> primitive nfs-kernel-server lsb:nfs-kernel-server \<br>
> >>>>>> op monitor interval="10s" timeout="30s"<br>
> >>>>>> primitive ping ocf:pacemaker:ping \<br>
> >>>>>> params host_list="10.236.22.35" multiplier="100" name="ping"<br>
> >>> \<br>
> >>>>>> op monitor interval="20s" timeout="60s" \<br>
> >>>>>> op start interval="0" timeout="60s"<br>
> >>>>>> primitive portmap upstart:portmap \<br>
> >>>>>> op monitor interval="10s" timeout="30s"<br>
> >>>>>> primitive res-dlm ocf:pacemaker:controld \<br>
> >>>>>> op monitor interval="120s"<br>
> >>>>>> primitive res-fs ocf:heartbeat:Filesystem \<br>
> >>>>>> params device="/dev/mapper/mpath0" directory="/media/media0"<br>
> >>>>>> fstype="ocfs2" \<br>
> >>>>>> op monitor interval="120s"<br>
> >>>>>> primitive res-nfs1-ip ocf:heartbeat:IPaddr2 \<br>
> >>>>>> params ip="10.236.22.38" cidr_netmask="27" nic="bond0" \<br>
> >>>>>> op monitor interval="30s"<br>
> >>>>>> primitive res-nfs2-ip ocf:heartbeat:IPaddr2 \<br>
> >>>>>> params ip="10.236.22.39" cidr_netmask="27" nic="bond0" \<br>
> >>>>>> op monitor interval="30s"<br>
> >>>>>> primitive res-o2cb ocf:pacemaker:o2cb \<br>
> >>>>>> op monitor interval="120s"<br>
> >>>>>> primitive res-share ocf:heartbeat:exportfs \<br>
> >>>>>> params directory="/media/media0/nfsroot/export1" clientspec="<br>
> >>>>>> <a href="http://10.236.22.0/24" target="_blank">10.236.22.0/24</a>" options="rw,async,no_subtree_check,no_root_squash"<br>
> >>>>> fsid="1"<br>
> >>>>>> \<br>
> >>>>>> op monitor interval="10s" timeout="30s" \<br>
> >>>>>> op start interval="10" timeout="40s" \<br>
> >>>>>> op stop interval="0" timeout="40s"<br>
> >>>>>> primitive st-null stonith:null \<br>
> >>>>>> params hostlist="msk-nfs-gw01 msk-nfs-gw02"<br>
> >>>>>> group nfs portmap nfs-kernel-server<br>
> >>>>>> clone clone-dlm res-dlm \<br>
> >>>>>> meta globally-unique="false" interleave="true"<br>
> >>>>>> clone clone-fs res-fs \<br>
> >>>>>> meta globally-unique="false" interleave="true"<br>
> >>>>>> clone clone-nfs nfs \<br>
> >>>>>> meta globally-unique="false" interleace="true"<br>
> >>>>>> clone clone-o2cb res-o2cb \<br>
> >>>>>> meta globally-unique="false" interleave="true"<br>
> >>>>>> clone clone-share res-share \<br>
> >>>>>> meta globally-unique="false" interleave="true"<br>
> >>>>>> clone fencing st-null<br>
> >>>>>> clone ping_clone ping \<br>
> >>>>>> meta globally-unique="false"<br>
> >>>>>> location nfs1-ip-on-nfs1 res-nfs1-ip 50: msk-nfs-gw01<br>
> >>>>>> location nfs2-ip-on-nfs2 res-nfs2-ip 50: msk-nfs-gw02<br>
> >>>>>> colocation col-fs-o2cb inf: clone-fs clone-o2cb<br>
> >>>>>> colocation col-nfs-fs inf: clone-nfs clone-fs<br>
> >>>>>> colocation col-o2cb-dlm inf: clone-o2cb clone-dlm<br>
> >>>>>> colocation col-share-nfs inf: clone-share clone-nfs<br>
> >>>>>> order ord-dlm-o2cb 0: clone-dlm clone-o2cb<br>
> >>>>>> order ord-nfs-share 0: clone-nfs clone-share<br>
> >>>>>> order ord-o2cb-fs 0: clone-o2cb clone-fs<br>
> >>>>>> order ord-o2cb-nfs 0: clone-fs clone-nfs<br>
> >>>>>> order ord-share-nfs1 0: clone-share res-nfs1-ip<br>
> >>>>>> order ord-share-nfs2 0: clone-share res-nfs2-ip<br>
> >>>>>> property $id="cib-bootstrap-options" \<br>
> >>>>>> dc-version="1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3" \<br>
> >>>>>> cluster-infrastructure="openais" \<br>
> >>>>>> expected-quorum-votes="2" \<br>
> >>>>>> stonith-enabled="true" \<br>
> >>>>>> no-quorum-policy="ignore" \<br>
> >>>>>> last-lrm-refresh="1308040111"<br>
> >>>>>><br>
> >>>>>> --<br>
> >>>>>> Best Regards<br>
> >>>>>> Alexander Malaev<br>
> >>>>><br>
> >>>>>> _______________________________________________<br>
> >>>>>> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> >>>>>> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> >>>>>><br>
> >>>>>> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> >>>>>> Getting started:<br>
> >>> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> >>>>>> Bugs:<br>
> >>>>><br>
> >>> <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
> >>>>><br>
> >>>>><br>
> >>>><br>
> >>>><br>
> >>>> --<br>
> >>>> С уважением,<br>
> >>>> Александр Малаев<br>
> >>>> +7-962-938-9323<br>
> >>><br>
> >>>> _______________________________________________<br>
> >>>> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> >>>> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> >>>><br>
> >>>> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> >>>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> >>>> Bugs:<br>
> >>> <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
> >>><br>
> >>><br>
> >><br>
> >><br>
> >> --<br>
> >> С уважением,<br>
> >> Александр Малаев<br>
> >> +7-962-938-9323<br>
> ><br>
> >> _______________________________________________<br>
> >> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> >> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> >><br>
> >> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> >> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> >> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> > <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> ><br>
> > Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> > Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>С уважением,<br>Александр Малаев<br>+7-962-938-9323<br>
</div>