[Pacemaker] Connection to our AIS plugin (9) failed: Library error
Szymon Hersztek
szym at globtel.pl
Tue Sep 21 03:59:05 EDT 2010
Wiadomość napisana w dniu 2010-09-21, o godz. 09:08, przez Andrew
Beekhof:
> 2010/9/21 Szymon Hersztek <szym at globtel.pl>:
>>
>> Wiadomość napisana w dniu 2010-09-21, o godz. 08:34, przez Andrew
>> Beekhof:
>>
>>> On Mon, Sep 20, 2010 at 3:34 PM, Szymon Hersztek <szym at globtel.pl>
>>> wrote:
>>>>
>>>> Hi
>>>> Im trying to setup corosync to work as drbd cluster but after
>>>> installing
>>>> follow by http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> i got error like below:
>>>
>>> Unusual, but did pacemaker fork a replacement attrd process?
>>> At what time did corosync start?
>>>
>>
>> corosync was started manually or do you want to have exact time of
>> start ?
>
> well you included at most 1 second's worth of logging.
> so its kinda hard to know if something took too long or what recovery
> was attempted.
Standard logs
Sep 21 09:57:31 drbd3 corosync[3015]: [MAIN ] Corosync Cluster
Engine ('1.2.7'): started and ready to provide service.
Sep 21 09:57:31 drbd3 corosync[3015]: [MAIN ] Corosync built-in
features: nss rdma
Sep 21 09:57:31 drbd3 corosync[3015]: [MAIN ] Successfully read
main configuration file '/etc/corosync/corosync.conf'.
Sep 21 09:57:31 drbd3 corosync[3015]: [TOTEM ] Initializing
transport (UDP/IP).
Sep 21 09:57:31 drbd3 corosync[3015]: [TOTEM ] Initializing transmit/
receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Sep 21 09:57:31 drbd3 corosync[3015]: [TOTEM ] The network interface
[192.168.1.212] is now up.
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info:
process_ais_conf: Reading configure
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info:
config_find_init: Local handle: 2013064636357672962 for logging
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info:
config_find_next: Processing additional logging options...
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Found 'off' for option: debug
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Found 'yes' for option: to_logfile
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Found '/var/log/cluster/corosync.log' for option: logfile
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Found 'yes' for option: to_syslog
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Defaulting to 'daemon' for option: syslog_facility
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info:
config_find_init: Local handle: 4730966301143465987 for service
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info:
config_find_next: Processing additional service options...
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Defaulting to 'pcmk' for option: clustername
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Defaulting to 'no' for option: use_logd
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: get_config_opt:
Defaulting to 'no' for option: use_mgmtd
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: pcmk_startup:
CRM: Initialized
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] Logging: Initialized
pcmk_startup
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: pcmk_startup:
Maximum core file size is: 18446744073709551615
Sep 21 09:57:31 drbd3 lrmd: [3023]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Sep 21 09:57:31 drbd3 corosync[3015]: [pcmk ] info: pcmk_startup:
Service: 9
Sep 21 09:57:32 drbd3 cib: [3022]: info: Invoked: /usr/lib64/heartbeat/
cib
Sep 21 09:57:32 drbd3 crmd: [3026]: info: Invoked: /usr/lib64/
heartbeat/crmd
Sep 21 09:57:32 drbd3 pengine: [3025]: info: Invoked: /usr/lib64/
heartbeat/pengine
Sep 21 09:57:32 drbd3 stonithd: [3021]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Sep 21 09:57:32 drbd3 attrd: [3024]: info: Invoked: /usr/lib64/
heartbeat/attrd
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: pcmk_startup:
Local hostname: drbd3.xxx.xxx
Sep 21 09:57:32 drbd3 cib: [3022]: info: G_main_add_TriggerHandler:
Added signal manual handler
Sep 21 09:57:32 drbd3 crmd: [3026]: info: main: CRM Hg Version:
89bd754939df5150de7cd76835f98fe90851b677
Sep 21 09:57:32 drbd3 lrmd: [3023]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Sep 21 09:57:32 drbd3 stonithd: [3021]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Sep 21 09:57:32 drbd3 pengine: [3025]: info: main: Starting pengine
Sep 21 09:57:32 drbd3 attrd: [3024]: info: main: Starting up
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info:
pcmk_update_nodeid: Local node id: 3556878528
Sep 21 09:57:32 drbd3 cib: [3022]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Sep 21 09:57:32 drbd3 cib: [3022]: info: retrieveCib: Reading cluster
configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/
heartbeat/crm/cib.xml.sig)
Sep 21 09:57:32 drbd3 lrmd: [3023]: info: enabling coredumps
Sep 21 09:57:32 drbd3 stonithd: [3021]: info: crm_cluster_connect:
Connecting to OpenAIS
Sep 21 09:57:32 drbd3 attrd: [3024]: info: crm_cluster_connect:
Connecting to OpenAIS
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Creating entry for node 3556878528 born on 0
Sep 21 09:57:32 drbd3 cib: [3022]: WARN: retrieveCib: Cluster
configuration not found: /var/lib/heartbeat/crm/cib.xml
Sep 21 09:57:32 drbd3 crmd: [3026]: info: crmd_init: Starting crmd
Sep 21 09:57:32 drbd3 lrmd: [3023]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Sep 21 09:57:32 drbd3 stonithd: [3021]: info:
init_ais_connection_once: Creating connection to our AIS plugin
Sep 21 09:57:32 drbd3 attrd: [3024]: info: init_ais_connection_once:
Creating connection to our AIS plugin
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
0x2aaab0000ad0 Node 3556878528 now known as drbd3.xxx.xxx (was: (null))
Sep 21 09:57:32 drbd3 cib: [3022]: WARN: readCibXmlFile: Primary
configuration corrupt or unusable, trying backup...
Sep 21 09:57:32 drbd3 crmd: [3026]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Sep 21 09:57:32 drbd3 lrmd: [3023]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Sep 21 09:57:32 drbd3 attrd: [3024]: info: init_ais_connection_once:
Connection to our AIS plugin (9) failed: Library error (2)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node drbd3.xxx.xxx now has 1 quorum votes (was 0)
Sep 21 09:57:32 drbd3 cib: [3022]: WARN: readCibXmlFile: Continuing
with an empty configuration.
Sep 21 09:57:32 drbd3 lrmd: [3023]: info: Started.
Sep 21 09:57:32 drbd3 attrd: [3024]: ERROR: main: HA Signon failed
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node 3556878528/drbd3.xxx.xxx is now: member
Sep 21 09:57:32 drbd3 stonithd: [3021]: info:
init_ais_connection_once: AIS connection established
Sep 21 09:57:32 drbd3 attrd: [3024]: info: main: Cluster connection
active
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: spawn_child:
Forked child 3021 for process stonithd
Sep 21 09:57:32 drbd3 stonithd: [3021]: info: get_ais_nodeid: Server
details: id=3556878528 uname=drbd3.xxx.xxx cname=pcmk
Sep 21 09:57:32 drbd3 attrd: [3024]: info: main: Accepting attribute
updates
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: spawn_child:
Forked child 3022 for process cib
Sep 21 09:57:32 drbd3 stonithd: [3021]: info: crm_new_peer: Node
drbd3.xxx.xxx now has id: 3556878528
Sep 21 09:57:32 drbd3 attrd: [3024]: ERROR: main: Aborting startup
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: spawn_child:
Forked child 3023 for process lrmd
Sep 21 09:57:32 drbd3 stonithd: [3021]: info: crm_new_peer: Node
3556878528 is now known as drbd3.xxx.xxx
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: spawn_child:
Forked child 3024 for process attrd
Sep 21 09:57:32 drbd3 stonithd: [3021]: notice: /usr/lib64/heartbeat/
stonithd start up successfully.
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: spawn_child:
Forked child 3025 for process pengine
Sep 21 09:57:32 drbd3 stonithd: [3021]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: spawn_child:
Forked child 3026 for process crmd
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: Pacemaker Cluster Manager 1.0.9
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: corosync extended virtual synchrony service
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: corosync configuration service
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: corosync cluster closed process group service v1.01
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: corosync cluster config database access v1.01
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: corosync profile loading service
Sep 21 09:57:32 drbd3 corosync[3015]: [SERV ] Service engine
loaded: corosync cluster quorum service v0.1
Sep 21 09:57:32 drbd3 corosync[3015]: [MAIN ] Compatibility mode
set to whitetank. Using V1 and V2 of the synchronization engine.
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] notice:
pcmk_peer_update: Transitional membership event on ring 4: memb=0,
new=0, lost=0
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] notice:
pcmk_peer_update: Stable membership event on ring 4: memb=1, new=1,
lost=0
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info:
pcmk_peer_update: NEW: drbd3.xxx.xxx 3556878528
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info:
pcmk_peer_update: MEMB: drbd3.xxx.xxx 3556878528
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node drbd3.xxx.xxx now has process list:
00000000000000000000000000013312 (78610)
Sep 21 09:57:32 drbd3 corosync[3015]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Sep 21 09:57:32 drbd3 corosync[3015]: [MAIN ] Completed service
synchronization, ready to provide service.
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: pcmk_ipc:
Recorded connection 0x1ed38ea0 for stonithd/3021
Sep 21 09:57:32 drbd3 cib: [3022]: info: startCib: CIB Initialization
completed successfully
Sep 21 09:57:32 drbd3 cib: [3022]: info: crm_cluster_connect:
Connecting to OpenAIS
Sep 21 09:57:32 drbd3 cib: [3022]: info: init_ais_connection_once:
Creating connection to our AIS plugin
Sep 21 09:57:32 drbd3 cib: [3022]: info: init_ais_connection_once:
Connection to our AIS plugin (9) failed: Library error (2)
Sep 21 09:57:32 drbd3 cib: [3022]: CRIT: cib_init: Cannot sign in to
the cluster... terminating
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] ERROR:
pcmk_wait_dispatch: Child process cib exited (pid=3022, rc=100)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] notice:
pcmk_wait_dispatch: Child process cib no longer wishes to be respawned
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node drbd3.xxx.xxx now has process list:
00000000000000000000000000013212 (78354)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] ERROR:
pcmk_wait_dispatch: Child process attrd exited (pid=3024, rc=100)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] notice:
pcmk_wait_dispatch: Child process attrd no longer wishes to be respawned
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node drbd3.xxx.xxx now has process list:
00000000000000000000000000012212 (74258)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node drbd3.xxx.xxx now has process list:
00000000000000000000000000013212 (78354)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info: update_member:
Node drbd3.xxx.xxx now has process list:
00000000000000000000000000012212 (74258)
Sep 21 09:57:32 drbd3 corosync[3015]: [pcmk ] info:
send_member_notification: Sending membership update 4 to 0 children
Sep 21 09:57:33 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:33 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 1 times... pause and retry
Sep 21 09:57:33 drbd3 crmd: [3026]: info: crmd_init: Starting crmd's
mainloop
Sep 21 09:57:35 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:36 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:36 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 2 times... pause and retry
Sep 21 09:57:38 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:39 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:39 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 3 times... pause and retry
Sep 21 09:57:41 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:42 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:42 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 4 times... pause and retry
Sep 21 09:57:44 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:45 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:45 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 5 times... pause and retry
Sep 21 09:57:47 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:48 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:48 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 6 times... pause and retry
Sep 21 09:57:50 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:51 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:51 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 7 times... pause and retry
Sep 21 09:57:53 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:54 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:54 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 8 times... pause and retry
Sep 21 09:57:56 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:57:57 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:57:57 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 9 times... pause and retry
Sep 21 09:57:59 drbd3 crmd: [3026]: info: crm_timer_popped: Wait Timer
(I_NULL) just popped!
Sep 21 09:58:00 drbd3 crmd: [3026]: info: do_cib_control: Could not
connect to the CIB service: connection failed
Sep 21 09:58:00 drbd3 crmd: [3026]: WARN: do_cib_control: Couldn't
complete CIB registration 10 times... pause and retry
More information about the Pacemaker
mailing list