[Pacemaker] OpenAIS + mgmtd [Not Working]

Tue Nov 18 06:08:54 UTC 2008

On Mon, 2008-11-17 at 13:59 -0700, Bret Palsson wrote:
> When I try to start mgmtd after starting OpenAIS this is the output  
> in /var/log/messages:
> 
> 
> ## /etc/init.d/openais start
> 
> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] AIS Executive Service  
> RELEASE 'subrev 1152 version 0.80'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] Copyright (C) 2002-2006  
> MontaVista Software, Inc and contributors.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] Copyright (C) 2006 Red  
> Hat, Inc.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] AIS Executive Service:  
> started and ready to provide service.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Token Timeout (10000 ms)  
> retransmit timeout (495 ms)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] token hold (386 ms)  
> retransmits before loss (20 retrans)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] join (60 ms) send_join  
> (0 ms) consensus (4800 ms) merge (200 ms)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] downcheck (1000 ms) fail  
> to recv const (50 msgs)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] seqno unchanged const  
> (30 rotations) Maximum network MTU 1500
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] window size per rotation  
> (50 messages) maximum messages per rotation (20 messages)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] send threads (0 threads)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP token expired  
> timeout (495 ms)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP token problem  
> counter (2000 ms)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP threshold (10  
> problem count)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP mode set to none.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM]  
> heartbeat_failures_allowed (0)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] max_network_delay (50 ms)
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] HeartBeat is Disabled.  
> To enable set heartbeat_failures_allowed > 0
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Receive multicast socket  
> recv buffer size (262142 bytes).
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Transmit multicast  
> socket send buffer size (262142 bytes).
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] The network interface  
> [10.128.6.3] is now up.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Created or loaded  
> sequence id 0.10.128.6.3 for this ring.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] entering GATHER state  
> from 15.
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais extended virtual synchrony service'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais cluster membership service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais availability management framework B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais checkpoint service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais distributed locking service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais message service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais configuration service'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais cluster closed process group service v1.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais cluster closed process group service v1.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais configuration service'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais message service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais distributed locking service B.01.01'
> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:48:00 m-nfs2 last message repeated 32610 times
> Nov 17 13:49:02 m-nfs2 last message repeated 13604 times
> Nov 17 13:50:03 m-nfs2 last message repeated 10338 times
> Nov 17 13:50:11 m-nfs2 last message repeated 1295 times
> 
> 
> ## ./usr/lib64/heartbeat/mgmtd
> 
> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: G_main_add_SignalHandler:  
> Added signal handler for signal 15
> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: G_main_add_SignalHandler:  
> Added signal handler for signal 10
> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: G_main_add_SignalHandler:  
> Added signal handler for signal 12
> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not  
> initiate connection
> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: login to lrm: 0, ret:0
> Nov 17 13:50:11 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:50:12 m-nfs2 last message repeated 150 times
> Nov 17 13:50:12 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not  
> initiate connection
> Nov 17 13:50:12 m-nfs2 mgmtd: [4160]: info: login to lrm: 1, ret:0
> Nov 17 13:50:12 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:50:13 m-nfs2 last message repeated 149 times
> Nov 17 13:50:13 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not  
> initiate connection
> Nov 17 13:50:13 m-nfs2 mgmtd: [4160]: info: login to lrm: 2, ret:0
> Nov 17 13:50:13 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:50:14 m-nfs2 last message repeated 150 times
> Nov 17 13:50:14 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not  
> initiate connection
> Nov 17 13:50:14 m-nfs2 mgmtd: [4160]: info: login to lrm: 3, ret:0
> Nov 17 13:50:14 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:50:15 m-nfs2 last message repeated 148 times
> Nov 17 13:50:15 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not  
> initiate connection
> Nov 17 13:50:15 m-nfs2 mgmtd: [4160]: info: login to lrm: 4, ret:0
> Nov 17 13:50:15 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> Nov 17 13:50:16 m-nfs2 last message repeated 149 times
> Nov 17 13:50:16 m-nfs2 mgmtd: [4160]: info: login to lrm failed
> Nov 17 13:50:16 m-nfs2 mgmtd: [4160]: ERROR: Can't initialize  
> management library.Shutting down.(-1)
> Nov 17 13:50:16 m-nfs2 openais[4111]: [SERV ] Service initialized  
> 'openais event service B.01.01'
> 
> 
> ## ./usr/lib64/heartbeat/mgmtdtest
> ##  can't conenct to mgmtd
> 
> Does anyone know what might be wrong here? I shouldn't have to run the  
> heartbeat stack when I am running the OpenAIS stack.
You shouldn't. 
It seems that something is wrong with openais. And several daemons (at
least lrmd) could not be started. 

I've just tested the latest build (on Nov 14th) of openais and met the
similar issue. 

BTW, once it's resovled, you should start mgmtd with openais as :
# HA_cluster_type="openais" /usr/lib64/heartbeat/mgmtd

Regards,
-- 
Yan Gao
China R&D Software Engineer
ygao at novell.com

Novell, Inc.
SUSE® Linux Enterprise 10
Your Linux is ready
http://www.novell.com/linux