[Pacemaker] OpenAIS + mgmtd [Not Working]

[Andrew Beekhof]

On Tue, Nov 18, 2008 at 07:08, Yan Gao <ygao at novell.com> wrote:
> On Mon, 2008-11-17 at 13:59 -0700, Bret Palsson wrote:
>> When I try to start mgmtd after starting OpenAIS this is the output
>> in /var/log/messages:
>>
>>
>> ## /etc/init.d/openais start
>>
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] AIS Executive Service
>> RELEASE 'subrev 1152 version 0.80'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] Copyright (C) 2002-2006
>> MontaVista Software, Inc and contributors.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] Copyright (C) 2006 Red
>> Hat, Inc.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [MAIN ] AIS Executive Service:
>> started and ready to provide service.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Token Timeout (10000 ms)
>> retransmit timeout (495 ms)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] token hold (386 ms)
>> retransmits before loss (20 retrans)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] join (60 ms) send_join
>> (0 ms) consensus (4800 ms) merge (200 ms)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] downcheck (1000 ms) fail
>> to recv const (50 msgs)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] seqno unchanged const
>> (30 rotations) Maximum network MTU 1500
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] window size per rotation
>> (50 messages) maximum messages per rotation (20 messages)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] send threads (0 threads)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP token expired
>> timeout (495 ms)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP token problem
>> counter (2000 ms)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP threshold (10
>> problem count)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] RRP mode set to none.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM]
>> heartbeat_failures_allowed (0)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] max_network_delay (50 ms)
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] HeartBeat is Disabled.
>> To enable set heartbeat_failures_allowed > 0
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Receive multicast socket
>> recv buffer size (262142 bytes).
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Transmit multicast
>> socket send buffer size (262142 bytes).
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] The network interface
>> [10.128.6.3] is now up.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] Created or loaded
>> sequence id 0.10.128.6.3 for this ring.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [TOTEM] entering GATHER state
>> from 15.
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais extended virtual synchrony service'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais cluster membership service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais availability management framework B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais checkpoint service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais distributed locking service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais message service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais configuration service'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais cluster closed process group service v1.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais cluster closed process group service v1.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais configuration service'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais message service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais distributed locking service B.01.01'
>> Nov 17 13:47:30 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:48:00 m-nfs2 last message repeated 32610 times
>> Nov 17 13:49:02 m-nfs2 last message repeated 13604 times
>> Nov 17 13:50:03 m-nfs2 last message repeated 10338 times
>> Nov 17 13:50:11 m-nfs2 last message repeated 1295 times
>>
>>
>> ## ./usr/lib64/heartbeat/mgmtd
>>
>> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 15
>> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 10
>> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 12
>> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not
>> initiate connection
>> Nov 17 13:50:11 m-nfs2 mgmtd: [4160]: info: login to lrm: 0, ret:0
>> Nov 17 13:50:11 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:50:12 m-nfs2 last message repeated 150 times
>> Nov 17 13:50:12 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not
>> initiate connection
>> Nov 17 13:50:12 m-nfs2 mgmtd: [4160]: info: login to lrm: 1, ret:0
>> Nov 17 13:50:12 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:50:13 m-nfs2 last message repeated 149 times
>> Nov 17 13:50:13 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not
>> initiate connection
>> Nov 17 13:50:13 m-nfs2 mgmtd: [4160]: info: login to lrm: 2, ret:0
>> Nov 17 13:50:13 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:50:14 m-nfs2 last message repeated 150 times
>> Nov 17 13:50:14 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not
>> initiate connection
>> Nov 17 13:50:14 m-nfs2 mgmtd: [4160]: info: login to lrm: 3, ret:0
>> Nov 17 13:50:14 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:50:15 m-nfs2 last message repeated 148 times
>> Nov 17 13:50:15 m-nfs2 mgmtd: [4160]: WARN: lrm_signon: can not
>> initiate connection
>> Nov 17 13:50:15 m-nfs2 mgmtd: [4160]: info: login to lrm: 4, ret:0
>> Nov 17 13:50:15 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>> Nov 17 13:50:16 m-nfs2 last message repeated 149 times
>> Nov 17 13:50:16 m-nfs2 mgmtd: [4160]: info: login to lrm failed
>> Nov 17 13:50:16 m-nfs2 mgmtd: [4160]: ERROR: Can't initialize
>> management library.Shutting down.(-1)
>> Nov 17 13:50:16 m-nfs2 openais[4111]: [SERV ] Service initialized
>> 'openais event service B.01.01'
>>
>>
>> ## ./usr/lib64/heartbeat/mgmtdtest
>> ##  can't conenct to mgmtd
>>
>> Does anyone know what might be wrong here? I shouldn't have to run the
>> heartbeat stack when I am running the OpenAIS stack.
> You shouldn't.
> It seems that something is wrong with openais. And several daemons (at
> least lrmd) could not be started.
>
> I've just tested the latest build (on Nov 14th) of openais and met the
> similar issue.

Any reason you didn't tell anyone?

What does your openais.conf look like?
Are you telling it to load the pacemaker service?

>
> BTW, once it's resovled, you should start mgmtd with openais as :
> # HA_cluster_type="openais" /usr/lib64/heartbeat/mgmtd

With 1.0.1 you'll be able to add
 use_mgmtd: yes

to the pacemaker service block and we'll start it automatically.