[Pacemaker] Solved ! Fwd: after update one node in crm is getting offline CentOS

Testuser SST fatcharly at gmx.de
Tue Jun 15 08:28:25 EDT 2010 

-------- Original-Nachricht --------
Datum: Tue, 15 Jun 2010 14:13:19 +0200
Von: "Testuser  SST" <fatcharly at gmx.de>
An: pacemaker at oss.clusterlabs.org
Betreff: [Pacemaker] after update one node in crm is getting offline CentOS

Hi,

I have just made an update from heartbeat 2.x to the latest pacemaker with heartbeat and corosync from the clusterlabs repo on a 2 node CentOS-Cluster. (uninstall the heartbeat rpm, yum install from the new repo)
The Cluster is holding one IP-resource. When I start the first node with "service heartbeat start" eveythings work fine, and I can launch an crm_standby -v off -U NODENAME without any problem. But when I start the second node, the first node is ok, but the second is after a while shuting down itself before I can even launch an crm_standby command.

Node which is up: susan.hnc3.lan
Node which is initialising the shutdown: salina.hnc3.lan

crm_config:

INFO: building help index
node $id="5ed513ff-7d45-4cbe-b7ef-51be8b5a66ef" susan.hnc3.lan \
        attributes standby="off"
node $id="6f1f209b-751c-4a57-bae2-b9874be248cf" salina.hnc3.lan \
        attributes standby="on"
primitive mySQL_IP ocf:heartbeat:IPaddr2 \
        params ip="192.168.18.151" cidr_netmask="24" \
        op monitor interval="30s"
property $id="cib-bootstrap-options" \
        dc-version="1.0.8-9881a7350d6182bae9e8e557cf20a3cc5dac3ee7" \
        cluster-infrastructure="Heartbeat" \
        stonith-enabled="false" \
        no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
        resource-stickiness="100"



the last lines of the ha-log of salina:

 Jun 15 13:47:50 salina.hnc3.lan cib: [28079]: info: crm_update_peer_proc: susan.hnc3.lan.ais is now online
Jun 15 13:47:50 salina.hnc3.lan cib: [28079]: info: crm_update_peer_proc: susan.hnc3.lan.crmd is now online
Jun 15 13:47:50 salina.hnc3.lan cib: [28079]: info: crm_update_peer: Node salina.hnc3.lan: id=0 state=member (new) addr=(null
) votes=-1 born=2 seen=2 proc=00000000000000000000000000000100
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (i
d=2)
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=2, lost=0 n_idx=
0, new_idx=0, old_idx=4
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: ccm_event_detail:  CURRENT: susan.hnc3.lan [nodeid=1, born=1]
Jun 15 13:47:50 salina.hnc3.lan cib: [28079]: info: crm_update_peer_proc: salina.hnc3.lan.ais is now online
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: ccm_event_detail:  CURRENT: salina.hnc3.lan [nodeid=0, born=2]
Jun 15 13:47:50 salina.hnc3.lan cib: [28079]: info: crm_update_peer_proc: salina.hnc3.lan.crmd is now online
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: ccm_event_detail:  NEW:     susan.hnc3.lan [nodeid=1, born=1]
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: ccm_event_detail:  NEW:     salina.hnc3.lan [nodeid=0, born=2]
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: crm_get_peer: Node susan.hnc3.lan now has id: 1
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: crm_update_peer: Node susan.hnc3.lan: id=1 state=member (new) addr=(null
) votes=-1 born=1 seen=2 proc=00000000000000000000000000000200
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: crm_update_peer_proc: susan.hnc3.lan.ais is now online
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: crm_update_peer: Node salina.hnc3.lan: id=0 state=member (new) addr=(nul
l) votes=-1 born=2 seen=2 proc=00000000000000000000000000000200
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: crm_update_peer_proc: salina.hnc3.lan.ais is now online
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: do_started: The local CRM is operational
Jun 15 13:47:50 salina.hnc3.lan crmd: [28083]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_
PENDING cause=C_FSA_INTERNAL origin=do_started ]
Jun 15 13:47:52 salina.hnc3.lan cib: [28079]: info: cib_process_diff: Diff 0.21.5 -> 0.21.6 not applied to 0.18.0: current "e
poch" is less than required
Jun 15 13:47:52 salina.hnc3.lan crmd: [28083]: info: update_dc: Set DC to susan.hnc3.lan (3.0.1)
Jun 15 13:47:52 salina.hnc3.lan cib: [28079]: info: cib_server_process_diff: Requesting re-sync from peer
Jun 15 13:47:52 salina.hnc3.lan cib: [28079]: WARN: cib_diff_notify: Local-only Change (client:crmd, call: 37): 0.0.0 (Applic
ation of an update diff failed, requesting a full refresh)
Jun 15 13:47:52 salina.hnc3.lan cib: [28079]: info: cib_replace_notify: Replaced: 0.18.0 -> 0.21.6 from susan.hnc3.lan
Jun 15 13:47:52 salina.hnc3.lan crmd: [28083]: info: update_attrd: Connecting to attrd...
Jun 15 13:47:52 salina.hnc3.lan crmd: [28083]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NO
T_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: find_hash_entry: Creating hash entry for terminate
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: find_hash_entry: Creating hash entry for shutdown
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_local_callback: Sending full refresh (origin=crmd)
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<nu
ll>)
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_perform_update: Delaying operation terminate=<null>: cib not conn
ected
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<nul
l>)
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from salina.hnc3.lan
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from salina.hnc3.lan
Jun 15 13:47:52 salina.hnc3.lan cib: [28092]: info: write_cib_contents: Wrote version 0.21.0 of the CIB to disk (digest: 807f
50e28ff0e448e9a6a3b82a879d1c)
Jun 15 13:47:52 salina.hnc3.lan cib: [28092]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/c
ib-19.raw
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: cib_connect: Connected to the CIB after 2 signon attempts
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: cib_connect: Sending full refresh
ll>)
l>)
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from salina.hnc3.lan
Jun 15 13:47:52 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from salina.hnc3.lan
50e28ff0e448e9a6a3b82a879d1c)
ib.7YiWtn (digest: /var/lib/heartbeat/crm/cib.R6wctE)
Jun 15 13:47:52 salina.hnc3.lan cib: [28079]: info: Managed write_cib_contents process 28092 exited with return code 0.
Jun 15 13:47:53 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from salina.hnc3.lan
Jun 15 13:47:53 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from salina.hnc3.lan
]/transient_attributes": ok (rc=0)
]/lrm": ok (rc=0)
Jun 15 13:47:55 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from susan.hnc3.lan
Jun 15 13:47:55 salina.hnc3.lan attrd: [28082]: info: find_hash_entry: Creating hash entry for probe_complete
Jun 15 13:47:55 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from susan.hnc3.lan
Jun 15 13:47:55 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: flush message from susan.hnc3.lan
 op=mySQL_IP_monitor_0 )
Jun 15 13:47:55 salina.hnc3.lan lrmd: [28080]: info: rsc:mySQL_IP:2: probe
Jun 15 13:47:55 salina.hnc3.lan lrmd: [28080]: WARN: Managed mySQL_IP:monitor process 28097 exited with return code 7.
pdate=8, confirmed=true) not running
 (true)
Jun 15 13:47:57 salina.hnc3.lan attrd: [28082]: info: attrd_perform_update: Sen
t update 13: probe_complete=true
Jun 15 13:48:01 salina.hnc3.lan heartbeat: [28067]: info: killing /usr/lib/heartbeat/crmd process group 28083 with signal 15
Jun 15 13:48:01 salina.hnc3.lan crmd: [28083]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Jun 15 13:48:01 salina.hnc3.lan crmd: [28083]: info: crm_shutdown: Requesting shutdown
Jun 15 13:48:01 salina.hnc3.lan crmd: [28083]: info: do_shutdown_req: Sending shutdown request to DC: susan.hnc3.lan
Jun 15 13:48:02 salina.hnc3.lan attrd: [28082]: info: attrd_ha_callback: Update relayed from susan.hnc3.lan
602481)
Jun 15 13:48:02 salina.hnc3.lan attrd: [28082]: info: attrd_perform_update: Sent update 16: shutdown=1276602481
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: handle_request: Shutting down
TOP cause=C_HA_MESSAGE origin=route_message ]
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: do_shutdown: All subsystems stopped, continuing
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: do_lrm_control: Disconnected from the LRM
Jun 15 13:48:03 salina.hnc3.lan ccm: [28078]: info: client (pid=28083) removed from ccm
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: do_cib_control: Disconnecting CIB
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: crmd_cib_connection_destroy: Connection to the CIB terminated...
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd
origin=do_stop ]
Jun 15 13:48:03 salina.hnc3.lan crmd: [28083]: info: do_exit: [crmd] stopped (0)
Jun 15 13:48:03 salina.hnc3.lan heartbeat: [28067]: info: killing /usr/lib/heartbeat/attrd process group 28082 with signal 15
Jun 15 13:48:03 salina.hnc3.lan attrd: [28082]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Jun 15 13:48:03 salina.hnc3.lan attrd: [28082]: info: attrd_shutdown: Exiting
Jun 15 13:48:03 salina.hnc3.lan attrd: [28082]: info: main: Exiting...
Jun 15 13:48:03 salina.hnc3.lan attrd: [28082]: info: attrd_cib_connection_destroy: Connection to the CIB terminated...
 15
Jun 15 13:48:03 salina.hnc3.lan stonithd: [28081]: notice: /usr/lib/heartbeat/stonithd normally quit.
15
Jun 15 13:48:03 salina.hnc3.lan lrmd: [28080]: info: lrmd is shutting down
Jun 15 13:48:03 salina.hnc3.lan heartbeat: [28067]: info: killing /usr/lib/heartbeat/cib process group 28079 with signal 15
Jun 15 13:48:03 salina.hnc3.lan cib: [28079]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated
Jun 15 13:48:03 salina.hnc3.lan cib: [28079]: info: cib_shutdown: Disconnected 0 clients
Jun 15 13:48:03 salina.hnc3.lan cib: [28079]: info: cib_process_disconnect: All clients disconnected...
Jun 15 13:48:03 salina.hnc3.lan cib: [28079]: info: initiate_exit: Sending disconnect notification to 2 peers...
Jun 15 13:48:04 salina.hnc3.lan cib: [28079]: info: cib_process_shutdown_req: Shutdown ACK from susan.hnc3.lan
Jun 15 13:48:04 salina.hnc3.lan cib: [28079]: info: terminate_cib: cib_process_shutdown_req: Disconnecting heartbeat
Jun 15 13:48:04 salina.hnc3.lan cib: [28079]: info: terminate_cib: Exiting...
'all' (origin=susan.hnc3.lan/susan.hnc3.lan/(null), version=0.0.0): ok (rc=0)
Jun 15 13:48:04 salina.hnc3.lan cib: [28079]: info: ha_msg_dispatch: Lost connection to heartbeat service.
Jun 15 13:48:04 salina.hnc3.lan cib: [28079]: info: main: Done
Jun 15 13:48:04 salina.hnc3.lan ccm: [28078]: info: client (pid=28079) removed from ccm
Jun 15 13:48:04 salina.hnc3.lan heartbeat: [28067]: info: killing /usr/lib/heartbeat/ccm process group 28078 with signal 15
Jun 15 13:48:04 salina.hnc3.lan ccm: [28078]: info: received SIGTERM, going to shut down
Jun 15 13:48:04 salina.hnc3.lan heartbeat: [28067]: info: client [/usr/lib/heartbeat/ipfail] is not running.
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: killing HBREAD process 28072 with signal 15
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: killing HBWRITE process 28073 with signal 15
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: killing HBREAD process 28074 with signal 15
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: killing HBFIFO process 28070 with signal 15
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: killing HBWRITE process 28071 with signal 15
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: Core process 28074 exited. 5 remaining
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: Core process 28072 exited. 4 remaining
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: Core process 28071 exited. 3 remaining
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: Core process 28070 exited. 2 remaining
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: Core process 28073 exited. 1 remaining
Jun 15 13:48:06 salina.hnc3.lan heartbeat: [28067]: info: salina.hnc3.lan Heartbeat shutdown complete.


Full logfile is attached

Any assistance would be much appreciated!

Kind Regards

f_c
-- 
GMX DSL: Internet-, Telefon- und Handy-Flat ab 19,99 EUR/mtl.  
Bis zu 150 EUR Startguthaben inklusive! http://portal.gmx.net/de/go/dsl

-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
-------------- next part --------------
A non-text attachment was scrubbed...
Name: salina_ha-log
Type: application/octet-stream
Size: 29366 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/pacemaker/attachments/20100615/f483215e/attachment-0002.obj>
-------------- next part --------------
_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker

More information about the Pacemaker mailing list