[Pacemaker] "ERROR: Wrong stack o2cb" when trying to start o2cb service in Pacemaker cluster

David Guyot david.guyot at europecamions-interactive.com
Fri Jun 22 05:14:07 EDT 2012


Hello.

Concerning dlm-pcmk, it's not available from backports, so I installed
it from stable; only ocfs2-tools-pacemaker are available and installed
from it.

I checked if /etc/init.d/ocfs2 and /etc/init.d/o2cb are removed from
/etc/rcX.d/*, and they are, so the system cannot boot them up by itself.
I also reconfigured DRBD resources using notify=true in each DRBD
master, then I reconfigured OCFS2 resources using these crm commands

primitive p_controld ocf:pacemaker:controld
primitive p_o2cb ocf:ocfs2:o2cb
group g_ocfs2mgmt p_controld p_o2cb
clone cl_ocfs2mgmt g_ocfs2mgmt meta interleave=true

root at Malastare:/home/david# crm configure show
node Malastare
node Vindemiatrix
primitive p_controld ocf:pacemaker:controld
primitive p_drbd_backupvi ocf:linbit:drbd \
    params drbd_resource="backupvi"
primitive p_drbd_pgsql ocf:linbit:drbd \
    params drbd_resource="postgresql"
primitive p_drbd_svn ocf:linbit:drbd \
    params drbd_resource="svn"
primitive p_drbd_www ocf:linbit:drbd \
    params drbd_resource="www"
primitive p_o2cb ocf:pacemaker:o2cb
primitive soapi-fencing-malastare stonith:external/ovh \
    params reversedns="ns208812.ovh.net"
primitive soapi-fencing-vindemiatrix stonith:external/ovh \
    params reversedns="ns235795.ovh.net"
group g_ocfs2mgmt p_controld p_o2cb
ms ms_drbd_backupvi p_drbd_backupvi \
    meta master-max="2" clone-max="2" notify="true"
ms ms_drbd_pgsql p_drbd_pgsql \
    meta master-max="2" clone-max="2" notify="true"
ms ms_drbd_svn p_drbd_svn \
    meta master-max="2" clone-max="2" notify="true"
ms ms_drbd_www p_drbd_www \
    meta master-max="2" clone-max="2" notify="true"
clone cl_ocfs2mgmt g_ocfs2mgmt \
    meta interleave="true"
location stonith-malastare soapi-fencing-malastare -inf: Malastare
location stonith-vindemiatrix soapi-fencing-vindemiatrix -inf: Vindemiatrix
property $id="cib-bootstrap-options" \
    dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
    cluster-infrastructure="openais" \
    expected-quorum-votes="2"

Unfortunately, the problem is still there :

root at Malastare:/home/david# crm_mon --one-shot -VroA
============
Last updated: Fri Jun 22 10:54:31 2012
Last change: Fri Jun 22 10:54:27 2012 via crm_shadow on Malastare
Stack: openais
Current DC: Malastare - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
2 Nodes configured, 2 expected votes
14 Resources configured.
============

Online: [ Malastare Vindemiatrix ]

Full list of resources:

 soapi-fencing-malastare    (stonith:external/ovh):    Started Vindemiatrix
 soapi-fencing-vindemiatrix    (stonith:external/ovh):    Started Malastare
 Master/Slave Set: ms_drbd_pgsql [p_drbd_pgsql]
     Masters: [ Malastare Vindemiatrix ]
 Master/Slave Set: ms_drbd_svn [p_drbd_svn]
     Masters: [ Malastare Vindemiatrix ]
 Master/Slave Set: ms_drbd_www [p_drbd_www]
     Masters: [ Malastare Vindemiatrix ]
 Master/Slave Set: ms_drbd_backupvi [p_drbd_backupvi]
     Masters: [ Malastare Vindemiatrix ]
 Clone Set: cl_ocfs2mgmt [g_ocfs2mgmt]
     Stopped: [ g_ocfs2mgmt:0 g_ocfs2mgmt:1 ]

Node Attributes:
* Node Malastare:
    + master-p_drbd_backupvi:0            : 10000    
    + master-p_drbd_pgsql:0               : 10000    
    + master-p_drbd_svn:0                 : 10000    
    + master-p_drbd_www:0                 : 10000    
* Node Vindemiatrix:
    + master-p_drbd_backupvi:1            : 10000    
    + master-p_drbd_pgsql:1               : 10000    
    + master-p_drbd_svn:1                 : 10000    
    + master-p_drbd_www:1                 : 10000    

Operations:
* Node Vindemiatrix:
   soapi-fencing-malastare: migration-threshold=1000000
    + (4) start: rc=0 (ok)
   p_drbd_pgsql:1: migration-threshold=1000000
    + (5) probe: rc=8 (master)
   p_drbd_svn:1: migration-threshold=1000000
    + (6) probe: rc=8 (master)
   p_drbd_www:1: migration-threshold=1000000
    + (7) probe: rc=8 (master)
   p_drbd_backupvi:1: migration-threshold=1000000
    + (8) probe: rc=8 (master)
   p_o2cb:1: migration-threshold=1000000
    + (10) probe: rc=5 (not installed)
* Node Malastare:
   soapi-fencing-vindemiatrix: migration-threshold=1000000
    + (4) start: rc=0 (ok)
   p_drbd_pgsql:0: migration-threshold=1000000
    + (5) probe: rc=8 (master)
   p_drbd_svn:0: migration-threshold=1000000
    + (6) probe: rc=8 (master)
   p_drbd_www:0: migration-threshold=1000000
    + (7) probe: rc=8 (master)
   p_drbd_backupvi:0: migration-threshold=1000000
    + (8) probe: rc=8 (master)
   p_o2cb:0: migration-threshold=1000000
    + (10) probe: rc=5 (not installed)

Failed actions:
    p_o2cb:1_monitor_0 (node=Vindemiatrix, call=10, rc=5,
status=complete): not installed
    p_o2cb:0_monitor_0 (node=Malastare, call=10, rc=5, status=complete):
not installed

Nevertheless, I noticed a strange error message in Corosync/Pacemaker logs :
Jun 22 10:54:25 Vindemiatrix lrmd: [24580]: info: RA output:
(p_controld:1:probe:stderr) dlm_controld.pcmk: no process found

This message was immediately followed by "Wrong stack" errors, and
because dlm_controld.pcmk seems to be Pacemaker DLM dæmon, I strongly
thinks these messages are related. Strangely, even if I have this dæmon
executable in /usr/sbin, it's not loaded by Pacemaker :
root at Vindemiatrix:/home/david# ls /usr/sbin/dlm_controld.pcmk
/usr/sbin/dlm_controld.pcmk
root at Vindemiatrix:/home/david# ps fax | grep pcmk
26360 pts/1    S+     0:00                          \_ grep pcmk

But, if I understood correctly, such process should be launched by DLM
resource, and as I have no error messages concerning launching such a
process whereas its executable is present, do you know where this
problem could come from?

Thank you in advance.

Kind regards.

PS: I'll have the next week off, so I won't be able to answer you
between this evening and the 2th of July.

Le 20/06/2012 17:39, Andreas Kurz a écrit :
> On 06/20/2012 03:49 PM, David Guyot wrote:
>> Actually, yes, I start DRBD manually, because this is currently a test
>> configuration which relies on OpenVPN for the communications between
>> these 2 nodes. I have no order and collocation constraints because I'm
>> discovering these software and trying to configure them step by step and
>> make resources work before ordering them (nevertheless, I just tried to
>> configure DLM/O2CB constraints, but they fail, apparently because they
>> are relying on O2CB, which causes the problem I wrote you about.) And I
>> have no OCFS2 mounts because I was on the assumption that OCFS2 wouldn't
>> mount partitions without O2CB and DLM, which seems to be right :
> In fact it won't work without constraints, even if you are only testing
> e.g. controld and o2cb must run on the same node (in fact on both nodes
> of course) and controld must run before o2cb.
>
> And the error message you showed in a previous mail:
>
> 2012/06/20_09:04:35 ERROR: Wrong stack o2cb
>
> ... implies, that you are already running the native ocfs2 cluster stack
> outside of pacemaker. You did a "/etc/init.d/ocfs2" stop before starting
> your cluster tests and it is still stopped? And if it is stopped, a
> cleanup of cl_ocfs2mgmt resource should start that resource ... if there
> are no more other errors.
>
> You installed dlm-pcmk and ocfs2-tools-pacemaker packages from backports?
>
>> root at Malastare:/home/david# crm_mon --one-shot -VroA
>> ============
>> Last updated: Wed Jun 20 15:32:50 2012
>> Last change: Wed Jun 20 15:28:34 2012 via crm_shadow on Malastare
>> Stack: openais
>> Current DC: Vindemiatrix - partition with quorum
>> Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
>> 2 Nodes configured, 2 expected votes
>> 14 Resources configured.
>> ============
>>
>> Online: [ Vindemiatrix Malastare ]
>>
>> Full list of resources:
>>
>>  soapi-fencing-malastare    (stonith:external/ovh):    Started Vindemiatrix
>>  soapi-fencing-vindemiatrix    (stonith:external/ovh):    Started Malastare
>>  Master/Slave Set: ms_drbd_ocfs2_pgsql [p_drbd_ocfs2_pgsql]
>>      Masters: [ Malastare Vindemiatrix ]
>>  Master/Slave Set: ms_drbd_ocfs2_backupvi [p_drbd_ocfs2_backupvi]
>>      Masters: [ Malastare Vindemiatrix ]
>>  Master/Slave Set: ms_drbd_ocfs2_svn [p_drbd_ocfs2_svn]
>>      Masters: [ Malastare Vindemiatrix ]
>>  Master/Slave Set: ms_drbd_ocfs2_www [p_drbd_ocfs2_www]
>>      Masters: [ Malastare Vindemiatrix ]
>>  Clone Set: cl_ocfs2mgmt [g_ocfs2mgmt]
>>      Stopped: [ g_ocfs2mgmt:0 g_ocfs2mgmt:1 ]
>>
>> Node Attributes:
>> * Node Vindemiatrix:
>>     + master-p_drbd_ocfs2_backupvi:1      : 10000    
>>     + master-p_drbd_ocfs2_pgsql:1         : 10000    
>>     + master-p_drbd_ocfs2_svn:1           : 10000    
>>     + master-p_drbd_ocfs2_www:1           : 10000    
>> * Node Malastare:
>>     + master-p_drbd_ocfs2_backupvi:0      : 10000    
>>     + master-p_drbd_ocfs2_pgsql:0         : 10000    
>>     + master-p_drbd_ocfs2_svn:0           : 10000    
>>     + master-p_drbd_ocfs2_www:0           : 10000    
>>
>> Operations:
>> * Node Vindemiatrix:
>>    p_drbd_ocfs2_pgsql:1: migration-threshold=1000000
>>     + (4) probe: rc=8 (master)
>>    p_drbd_ocfs2_backupvi:1: migration-threshold=1000000
>>     + (5) probe: rc=8 (master)
>>    p_drbd_ocfs2_svn:1: migration-threshold=1000000
>>     + (6) probe: rc=8 (master)
>>    p_drbd_ocfs2_www:1: migration-threshold=1000000
>>     + (7) probe: rc=8 (master)
>>    soapi-fencing-malastare: migration-threshold=1000000
>>     + (10) start: rc=0 (ok)
>>    p_o2cb:1: migration-threshold=1000000
>>     + (9) probe: rc=5 (not installed)
>> * Node Malastare:
>>    p_drbd_ocfs2_pgsql:0: migration-threshold=1000000
>>     + (4) probe: rc=8 (master)
>>    p_drbd_ocfs2_backupvi:0: migration-threshold=1000000
>>     + (5) probe: rc=8 (master)
>>    p_drbd_ocfs2_svn:0: migration-threshold=1000000
>>     + (6) probe: rc=8 (master)
>>    soapi-fencing-vindemiatrix: migration-threshold=1000000
>>     + (10) start: rc=0 (ok)
>>    p_drbd_ocfs2_www:0: migration-threshold=1000000
>>     + (7) probe: rc=8 (master)
>>    p_o2cb:0: migration-threshold=1000000
>>     + (9) probe: rc=5 (not installed)
>>
>> Failed actions:
>>     p_o2cb:1_monitor_0 (node=Vindemiatrix, call=9, rc=5,
>> status=complete): not installed
>>     p_o2cb:0_monitor_0 (node=Malastare, call=9, rc=5, status=complete):
>> not installed
>> root at Malastare:/home/david# mount -t ocfs2 /dev/drbd1 /media/ocfs/
>> mount.ocfs2: Cluster stack specified does not match the one currently
>> running while trying to join the group
>>
>> Concerning the notify meta-attribute, I didn't configured it because it
>> wasn't even referred to in the DRBD official guide (
>> http://www.drbd.org/users-guide-8.3/s-ocfs2-pacemaker.html), and I don't
>> know what it does, so, by default, I stupidly followed the official
>> guide. What does this meta-attribute sets? If you know a better guide,
>> could you please tell me about, so I can check my config based on this
>> other guide?
> Well, than this is a documentation bug ... you will find the correct
> configuration in the same guide, where pacemaker integration is
> described ... "notify" sends out notification messages before and after
> an instance of the DRBD OCF RA exectutes an action (like start, stop,
> promote, demote) ... that allows the other instances to react.
>
> Regards,
> Andreas
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20120622/6f40375f/attachment-0003.sig>


More information about the Pacemaker mailing list