<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>Am 18.06.2014 um 01:13 schrieb Andrew Beekhof <<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>>:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br>On 18 Jun 2014, at 2:59 am, Krause, Markus <<a href="mailto:krause@biochem.mpg.de">krause@biochem.mpg.de</a>> wrote:<br><br><blockquote type="cite">Hi all,<span class="Apple-converted-space"> </span><br><br>I am using pacemaker/openais and drbd to have a high-available MySQL server which worked for years (sind SLES 11 SP 0) without configuration chances.<span class="Apple-converted-space"> </span><br>Just two days ago I did an update (zypper up) which now leads to the issue described below.<br>Although the subject of my email is quite similar to many others on this list I could not find an equivalent (so it seems) problem or a solution. I tried to provide as much information as seemed to be reasonable to me so I apologize in advance for the many lines!<br></blockquote><br>This looks ominous:<br><br> error: reap_dead_nodes:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>We're not part of the cluster anymore<br><br>It could be a software bug, but I'd start by checking the firewall.<br>Upgrades will sometimes affect those.<br></div></blockquote><div><br></div><div>Hi Andrew, </div><div><br></div><div>thanks for your response.</div><div>I already turned off the firewall, currently on both nodes iptables -nvL shows (here for sql01a):</div><div></div><div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"><span style="color: #760b00"><b>sql01a:~ # </b></span>iptables -nvL</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);">Chain INPUT (policy ACCEPT 0 packets, 0 bytes)</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"> pkts bytes target prot opt in out source destination </div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62); min-height: 10px;"><br></div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);">Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"> pkts bytes target prot opt in out source destination </div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62); min-height: 10px;"><br></div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);">Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"> pkts bytes target prot opt in out source destination </div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(118, 11, 0); background-color: rgb(19, 119, 62);"><b>sql01a:~ # </b></div><div></div></div><div><div><br></div><div>regards,</div><div> Markus</div></div><br><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br><blockquote type="cite"><br>Two servers with identical configuration (hard and software) running SLES 11 SP3 (with HAE) have the following crm config:<br><br><br>sql01a:~ # crm configure show<br>node sql01a<br>node sql01b<br>primitive drbd_drbd0 ocf:linbit:drbd \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params drbd_resource="drbd0" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="14" role="Master" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="16" role="Slave" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="240" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op stop interval="0" timeout="240"<br>primitive ha_ip ocf:heartbeat:IPaddr2 \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="5" timeout="20" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params ip="10.4.11.5" cidr_netmask="16" iflabel="ha" nic="eth0"<br>primitive mount_drbd0 ocf:heartbeat:Filesystem \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params device="/dev/drbd0" fstype="ext3" directory="/var/drbd0" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>meta target-role="Started" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="60" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op stop interval="0" timeout="300"<br>primitive mysqld ocf:heartbeat:mysql \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params binary="/usr/sbin/mysqld" config="/etc/my.cnf" datadir="/var/drbd0/mysql/data" user="mysql" group="mysql" log="/var/log/mysql.log" pid="/var/lib/mysql/mysqld.pid" socket="/var/lib/mysql/mysql.sock" test_table="ha_check.mysql_check" test_user=XXX" test_passwd=XXX" enable_creation="no" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="10" timeout="60" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="240" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op stop interval="0" timeout="240"<br>primitive ping_gateway ocf:pacemaker:ping \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params dampen="10" multiplier="100" host_list="10.4.0.1" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="60" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="5" timeout="60"<br>group group_mysqld mount_drbd0 mysqld ha_ip<br>ms ms_drbd_drbd0 drbd_drbd0 \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"<br>clone ping_clone ping_gateway<br>location loc_drbdmaster_drbd0 ms_drbd_drbd0 \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>rule $id="loc_drbdmaster_ping-rule" $role="Master" pingd: defined pingd<br>colocation col_ms_drbd_drbd0_mount_drbd0 inf: group_mysqld ms_drbd_drbd0:Master<br>order ord_ms_drbd_drbd0_mount_drbd0 inf: ms_drbd_drbd0:promote group_mysqld:start<br>property $id="cib-bootstrap-options" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>dc-version="1.1.10-f3eeaf4" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>cluster-infrastructure="classic openais (with plugin)" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>expected-quorum-votes="2" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>no-quorum-policy="ignore" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>stonith-enabled="false<br><br><br>this worked since the first setup long ago.<br>after the update (zypper up) two days ago only sql01b is running, sql01a is unable to join the cluster.<br><br><br>sql01a:~ # crm status:<br>Last updated: Tue Jun 17 18:20:29 2014<br>Last change: Tue Jun 17 17:26:20 2014 by root via crm_attribute on sql01b<br>Stack: classic openais (with plugin)<br>Current DC: sql01b - partition with quorum<br>Version: 1.1.10-f3eeaf4<br>2 Nodes configured, 2 expected votes<br>7 Resources configured<br><br><br>Online: [ sql01b ]<br>OFFLINE: [ sql01a ]<br><br>Clone Set: ping_clone [ping_gateway]<br> Started: [ sql01b ]<br> Stopped: [ sql01a ]<br>Master/Slave Set: ms_drbd_drbd0 [drbd_drbd0]<br> Masters: [ sql01b ]<br> Stopped: [ sql01a ]<br>Resource Group: group_mysqld<br> mount_drbd0<span class="Apple-tab-span" style="white-space: pre;"> </span>(ocf::heartbeat:Filesystem):<span class="Apple-tab-span" style="white-space: pre;"> </span>Started sql01b<span class="Apple-converted-space"> </span><br> mysqld<span class="Apple-tab-span" style="white-space: pre;"> </span>(ocf::heartbeat:mysql):<span class="Apple-tab-span" style="white-space: pre;"> </span>Started sql01b<span class="Apple-converted-space"> </span><br> ha_ip<span class="Apple-tab-span" style="white-space: pre;"> </span>(ocf::heartbeat:IPaddr2):<span class="Apple-tab-span" style="white-space: pre;"> </span>Started sql01b<br><br><br>Starting the cluster tools on sql01a (rcopenais start) leads to the following lines (snipped the OK-looking lines at top as I enabled debugging in corosync.conf; if necessary i can of course post the whole output, but thats a lot of lines ...) in /var/log/messages:<br><br><span class="Apple-converted-space"> </span><br>[snip]<br>Jun 17 18:20:18 sql01a corosync[14051]: [MAIN ] main.c:278 Completed service synchronization, ready to provide service.<br>Jun 17 18:20:18 sql01a corosync[14051]: [TOTEM ] totempg.c:292 waiting_trans_ack changed to 0<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a mgmtd: [14062]: debug: main: run the loop...<br>Jun 17 18:20:18 sql01a mgmtd: [14062]: info: Started.<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14056, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process cib no longer wishes to be respawned<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] utils.c:340 info: update_member: Node sql01a now has process list: 00000000000000000000000000151212 (1380882)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process crmd exited (pid=14061, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process crmd no longer wishes to be respawned<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] utils.c:340 info: update_member: Node sql01a now has process list: 00000000000000000000000000151012 (1380370)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process attrd exited (pid=14059, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process attrd no longer wishes to be respawned<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] utils.c:340 info: update_member: Node sql01a now has process list: 00000000000000000000000000150012 (1376274)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process pengine exited (pid=14060, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process pengine no longer wishes to be respawned<br>[snip]<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1365 info: send_member_notification: Sending membership update 1452 to 1 children<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:962 debug: pcmk_cluster_id_callback: Node update: sql01a (1.1.10)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:962 debug: pcmk_cluster_id_callback: Node update: sql01a (1.1.10)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:22 sql01a sshd[14068]: Connection closed by 10.4.1.60 [preauth]<br>Jun 17 18:20:23 sql01a corosync[14051]: [CONFDB] confdb.c:405 lib_init_fn: conn=0x663570<br>Jun 17 18:20:23 sql01a corosync[14051]: [CONFDB] confdb.c:411 exit_fn for conn=0x663570<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: error: setup_cib: Could not connect to the CIB service: Transport endpoint is not connected (-107)<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: crm_update_peer_state: plugin_handle_membership: Node sql01a[17499146] - state is now member (was (null))<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: get_node_name: Could not obtain a node name for classic openais (with plugin) nodeid 34276362<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: crm_update_peer_state: plugin_handle_membership: Node (null)[34276362] - state is now member (was (null))<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: plugin_handle_membership: Membership 1452: quorum acquired<br>[snip]<br><br><br>some more info:<br><br>sql01a:~ # corosync-cfgtool -s<br>Printing ring status.<br>Local node ID 17499146<br>RING ID 0<br><span class="Apple-tab-span" style="white-space: pre;"> </span>id<span class="Apple-tab-span" style="white-space: pre;"> </span>= 10.4.11.1<br><span class="Apple-tab-span" style="white-space: pre;"> </span>status<span class="Apple-tab-span" style="white-space: pre;"> </span>= ring 0 active with no faults<br>RING ID 1<br><span class="Apple-tab-span" style="white-space: pre;"> </span>id<span class="Apple-tab-span" style="white-space: pre;"> </span>= 10.255.4.1<br><span class="Apple-tab-span" style="white-space: pre;"> </span>status<span class="Apple-tab-span" style="white-space: pre;"> </span>= ring 1 active with no faults<br>sql01a:~ # corosync-objctl | grep member<br>runtime.totem.pg.mrp.srp.members.17499146.ip=r(0) ip(10.4.11.1) r(1) ip(10.255.4.1)<span class="Apple-converted-space"> </span><br>runtime.totem.pg.mrp.srp.members.17499146.join_count=1<br>runtime.totem.pg.mrp.srp.members.17499146.status=joined<br>runtime.totem.pg.mrp.srp.members.34276362.ip=r(0) ip(10.4.11.2) r(1) ip(10.255.4.2)<span class="Apple-converted-space"> </span><br>runtime.totem.pg.mrp.srp.members.34276362.join_count=1<br>runtime.totem.pg.mrp.srp.members.34276362.status=joined<br><br>This is similar to the other (working) host.<br><br>Interesting is:<br><br>sql01a:~ # rcopenais status<br>Running<br>sql01a:~ # ps ax | grep -E "pace|heart|coro" | grep -v grep<br>14051 ? Ssl 0:00 /usr/sbin/corosync<br>14057 ? S< 0:00 /usr/lib64/pacemaker/stonithd<br>14058 ? S< 0:00 /usr/lib64/pacemaker/lrmd<br>14062 ? S< 0:00 /usr/lib64/heartbeat/mgmtd<br>sql01a:~ #<span class="Apple-converted-space"> </span><br><br><br><br>For digging further whats going wrong I then started the missing processes (cib, attrd, pengine and crmd) from the command line (using the -V switch). cib, attrd and pengine are starting fine (even crm now shows a result, although wrong) but crmd fails:<br><br><br>sql01a:~ # /usr/lib64/pacemaker/crmd -V<br>notice: main:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>CRM Git Version: f3eeaf4<br> info: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_STARTUP from crmd_init() received in state S_STARTING<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_clm' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_evt' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_ckpt' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_amf_v2' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_msg' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_lck' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_tmr' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'pacemaker' for option: name<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found '0' for option: ver<br> info: get_cluster_type:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Detected an active 'classic openais (with plugin)' cluster<br> info: do_cib_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>CIB connection established<br>notice: crm_cluster_connect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connecting to cluster infrastructure: classic openais (with plugin)<br> info: init_cs_connection_classic:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Creating connection to our Corosync plugin<br> info: init_cs_connection_classic:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>AIS connection established<br>notice: get_node_name:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Defaulting to uname -n for the local classic openais (with plugin) node name<br> info: plugin_get_details:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Server details: id=17499146 uname=sql01a cname=pcmk<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Created entry 6002e998-b8ba-4daa-9c5d-1082bd1ab73a/0x65b8a0 for node sql01a/17499146 (1 total)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 17499146 is now known as sql01a<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01a is now (null)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 17499146 has uuid sql01a<br> info: crm_update_peer_proc:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>init_cs_connection_classic: Node sql01a[17499146] - unknown is now online<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Client sql01a/peer now has status [online] (DC=<null>)<br> info: init_cs_connection_once:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connection to 'classic openais (with plugin)': established<br>notice: get_node_name:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Defaulting to uname -n for the local classic openais (with plugin) node name<br> info: do_ha_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connected to the cluster<br> info: lrmd_ipc_connect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connecting to lrmd<br> info: do_lrm_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>LRM connection established<br> info: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Delaying start, no membership data (0000000000100000)<br> info: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Delaying start, no membership data (0000000000100000)<br> info: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Delaying start, no membership data (0000000000100000)<br>notice: plugin_handle_membership:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Membership 1460: quorum acquired<br> info: crm_update_peer_proc:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01a[17499146] - unknown is now member<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Client sql01a/peer now has status [offline] (DC=<null>)<br>notice: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Our peer connection failed<br>notice: crm_update_peer_state:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01a[17499146] - state is now member (was (null))<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01a is now member (was (null))<br> info: crm_update_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01a: id=17499146 state=member addr=r(0) ip(10.4.11.1) r(1) ip(10.255.4.1) (new) votes=1 (new) born=1460 seen=1460 proc=00000000000000000000000000140012<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Created entry 8925fb86-2b70-4c91-af03-a8251a9182fc/0x65ecf0 for node sql01b/34276362 (2 total)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 34276362 is now known as sql01b<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01b is now (null)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 34276362 has uuid sql01b<br> info: crm_update_peer_proc:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01b[34276362] - unknown is now member<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Client sql01b/peer now has status [online] (DC=<null>)<br>notice: crm_update_peer_state:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01b[34276362] - state is now member (was (null))<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01b is now member (was (null))<br> info: crm_update_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01b: id=34276362 state=member addr=r(0) ip(10.4.11.2) r(1) ip(10.255.4.2) (new) votes=1 (new) born=1436 seen=1460 proc=00000000000000000000000000151312<br> error: reap_dead_nodes:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>We're not part of the cluster anymore<br> info: plugin_handle_membership:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Membership 1460: quorum retained<br> error: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_ERROR from peer_update_callback() received in state S_STARTING<br>notice: do_state_transition:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>State transition S_STARTING -> S_RECOVERY [ input=I_ERROR cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]<br>warning: do_recover:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Fast-tracking shutdown in response to errors<br> error: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Start cancelled... S_RECOVERY<br> error: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_ERROR from reap_dead_nodes() received in state S_RECOVERY<br> info: do_dc_release:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>DC role released<br> info: do_te_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Transitioner is now inactive<br> error: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY<br> info: do_state_transition:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ]<br>notice: lrm_state_verify_stopped:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Stopped 0 recurring operations at shutdown (0 ops remaining)<br> info: do_lrm_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from the LRM<br> info: lrmd_api_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from lrmd service<br> info: lrmd_ipc_connection_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>IPC connection destroyed<br> info: lrm_connection_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>LRM Connection disconnected<br> info: lrmd_api_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from lrmd service<br>notice: do_lrm_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnected from the LRM<br> info: crm_cluster_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from cluster infrastructure: classic openais (with plugin)<br>notice: terminate_cs_connection:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from Corosync<br> info: cluster_disconnect_cpg:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>No CPG connection<br> info: crm_cluster_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnected from classic openais (with plugin)<br> info: do_ha_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnected from the cluster<br> info: do_cib_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting CIB<br> info: crmd_cib_connection_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connection to the CIB terminated...<br> info: do_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Performing A_EXIT_0 - gracefully exiting the CRMd<br> info: do_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>[crmd] stopped (0)<br> info: crmd_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Dropping I_RELEASE_SUCCESS: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_dc_release ]<br> info: crmd_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ]<br> info: crmd_cs_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>connection closed<br> info: crmd_init:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>13510 stopped: OK (0)<br> error: crmd_fast_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Could not recover from internal error<br> info: crm_xml_cleanup:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Cleaning up memory from libxml2<br>sql01a:~ #<span class="Apple-converted-space"> </span><br><br><br>what I tried up to now:<br>- stop sql01a, remove all cib.xml and crm.xml in their folders and restart sql01a (I know this is quite harsh but what did I have to loose
) -> the config on sql01a has been taken from sql01b but the crmd always dies as before (same error messages)<br>- I removed all cluster related packages (pacemaker, corosync, drbd,
) from sql01a and deleted the remaining dirs and config files, reinstalled all with copies from sql01b -> same result.<br>- I removed the whole crm config from sql01b (the working cluster host) using cibadmin -Q force and re-added the config (using crm commands") -> same result<br>- of course rebooted/restarted both servers several times -> nothing changes
<br><br><br>As deleting the config from sql01a (several times) I thought the error must be in the config/communication on/with sql01b but I also removed and added the config there, in addition there is now firewall between them and there is communication.<br><br>Does anybody know/understand what is going on here?<br>How can I solve this issue and get my cluster back to work as it did for quite a long time recently?<br><br><br>thanks in advance for any hints!<br><br>best regards,<br> Markus<br><br><br><br><br>-- .- .-. -.- ..- ... / -.- .-. .- ..- ... .<br>Markus Krause<br>Max Planck Institute of Biochemistry<br>- Computing Center (core services, networking, OS X)<br>Am Klopferspitz 18<br>D-82152 Martinsried, Germany<br>Phone: +49(89)8578-2825<br><br>_______________________________________________<br>Pacemaker mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><br>Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a><br></blockquote><br>_______________________________________________<br>Pacemaker mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><br>Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a></div></blockquote></div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px;"><font class="Apple-style-span" color="#033EFC">-- .- .-. -.- ..- ... / -.- .-. .- ..- ... .<br>Markus Krause<br>Max Planck Institute of Biochemistry<br>- Computing Center (core services, networking, OS X)<br>Am Klopferspitz 18<br>D-82152 Martinsried, Germany<br>Phone: +49(89)8578-2825</font></span></font></div></div>
</div>
<br></body></html>