<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>Am 18.06.2014 um 10:17 schrieb Krause, Markus <<a href="mailto:krause@biochem.mpg.de">krause@biochem.mpg.de</a>>:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div><br class="Apple-interchange-newline">Am 18.06.2014 um 01:13 schrieb Andrew Beekhof <<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>>:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br>On 18 Jun 2014, at 2:59 am, Krause, Markus <<a href="mailto:krause@biochem.mpg.de">krause@biochem.mpg.de</a>> wrote:<br><br><blockquote type="cite">Hi all,<span class="Apple-converted-space"> </span><br><br>I am using pacemaker/openais and drbd to have a high-available MySQL server which worked for years (sind SLES 11 SP 0) without configuration chances.<span class="Apple-converted-space"> </span><br>Just two days ago I did an update (zypper up) which now leads to the issue described below.<br>Although the subject of my email is quite similar to many others on this list I could not find an equivalent (so it seems) problem or a solution. I tried to provide as much information as seemed to be reasonable to me so I apologize in advance for the many lines!<br></blockquote><br>This looks ominous:<br><br> error: reap_dead_nodes:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>We're not part of the cluster anymore<br><br>It could be a software bug, but I'd start by checking the firewall.<br>Upgrades will sometimes affect those.<br></div></blockquote><div><br></div><div>Hi Andrew, </div><div><br></div><div>thanks for your response.</div><div>I already turned off the firewall, currently on both nodes iptables -nvL shows (here for sql01a):</div><div></div><div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"><span style="color: rgb(118, 11, 0);"><b>sql01a:~ #<span class="Apple-converted-space"> </span></b></span>iptables -nvL</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);">Chain INPUT (policy ACCEPT 0 packets, 0 bytes)</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"> pkts bytes target prot opt in out source destination </div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62); min-height: 10px;"><br></div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);">Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"> pkts bytes target prot opt in out source destination </div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62); min-height: 10px;"><br></div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);">Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)</div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(255, 240, 165); background-color: rgb(19, 119, 62);"> pkts bytes target prot opt in out source destination </div><div style="margin: 0px; font-size: 9px; font-family: Menlo; color: rgb(118, 11, 0); background-color: rgb(19, 119, 62);"><b>sql01a:~ # </b></div><div></div></div><div><div><br></div><div>regards,</div><div> Markus</div></div></div></blockquote><div><br></div><div>Hi all, </div><div>as I noticed that some permissions on the first node (sql01a) looked weird I decided to reinstall the whole system on this node and after copying all the configuration from the second host everything is working as expected without further changes. It seems that the update process messed up some permissions (and more?).</div><div><br></div><div>Thanks to all who answered to this issue for their time an assistance!</div><div><br></div><div>BTW: Is there a clean" way to remove all cluster relevant settings (deleting dirs, configurations) apart from reinstalling all if something is wrong on only one host?</div><div><br></div><div>regards,</div><div> Markus</div><div><br></div><br><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br><blockquote type="cite"><br>Two servers with identical configuration (hard and software) running SLES 11 SP3 (with HAE) have the following crm config:<br><br><br>sql01a:~ # crm configure show<br>node sql01a<br>node sql01b<br>primitive drbd_drbd0 ocf:linbit:drbd \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params drbd_resource="drbd0" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="14" role="Master" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="16" role="Slave" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="240" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op stop interval="0" timeout="240"<br>primitive ha_ip ocf:heartbeat:IPaddr2 \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="5" timeout="20" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params ip="10.4.11.5" cidr_netmask="16" iflabel="ha" nic="eth0"<br>primitive mount_drbd0 ocf:heartbeat:Filesystem \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params device="/dev/drbd0" fstype="ext3" directory="/var/drbd0" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>meta target-role="Started" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="60" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op stop interval="0" timeout="300"<br>primitive mysqld ocf:heartbeat:mysql \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params binary="/usr/sbin/mysqld" config="/etc/my.cnf" datadir="/var/drbd0/mysql/data" user="mysql" group="mysql" log="/var/log/mysql.log" pid="/var/lib/mysql/mysqld.pid" socket="/var/lib/mysql/mysql.sock" test_table="ha_check.mysql_check" test_user=XXX" test_passwd=XXX" enable_creation="no" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="10" timeout="60" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="240" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op stop interval="0" timeout="240"<br>primitive ping_gateway ocf:pacemaker:ping \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>params dampen="10" multiplier="100" host_list="10.4.0.1" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op start interval="0" timeout="60" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>op monitor interval="5" timeout="60"<br>group group_mysqld mount_drbd0 mysqld ha_ip<br>ms ms_drbd_drbd0 drbd_drbd0 \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"<br>clone ping_clone ping_gateway<br>location loc_drbdmaster_drbd0 ms_drbd_drbd0 \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>rule $id="loc_drbdmaster_ping-rule" $role="Master" pingd: defined pingd<br>colocation col_ms_drbd_drbd0_mount_drbd0 inf: group_mysqld ms_drbd_drbd0:Master<br>order ord_ms_drbd_drbd0_mount_drbd0 inf: ms_drbd_drbd0:promote group_mysqld:start<br>property $id="cib-bootstrap-options" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>dc-version="1.1.10-f3eeaf4" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>cluster-infrastructure="classic openais (with plugin)" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>expected-quorum-votes="2" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>no-quorum-policy="ignore" \<br><span class="Apple-tab-span" style="white-space: pre;"> </span>stonith-enabled="false<br><br><br>this worked since the first setup long ago.<br>after the update (zypper up) two days ago only sql01b is running, sql01a is unable to join the cluster.<br><br><br>sql01a:~ # crm status:<br>Last updated: Tue Jun 17 18:20:29 2014<br>Last change: Tue Jun 17 17:26:20 2014 by root via crm_attribute on sql01b<br>Stack: classic openais (with plugin)<br>Current DC: sql01b - partition with quorum<br>Version: 1.1.10-f3eeaf4<br>2 Nodes configured, 2 expected votes<br>7 Resources configured<br><br><br>Online: [ sql01b ]<br>OFFLINE: [ sql01a ]<br><br>Clone Set: ping_clone [ping_gateway]<br> Started: [ sql01b ]<br> Stopped: [ sql01a ]<br>Master/Slave Set: ms_drbd_drbd0 [drbd_drbd0]<br> Masters: [ sql01b ]<br> Stopped: [ sql01a ]<br>Resource Group: group_mysqld<br> mount_drbd0<span class="Apple-tab-span" style="white-space: pre;"> </span>(ocf::heartbeat:Filesystem):<span class="Apple-tab-span" style="white-space: pre;"> </span>Started sql01b<span class="Apple-converted-space"> </span><br> mysqld<span class="Apple-tab-span" style="white-space: pre;"> </span>(ocf::heartbeat:mysql):<span class="Apple-tab-span" style="white-space: pre;"> </span>Started sql01b<span class="Apple-converted-space"> </span><br> ha_ip<span class="Apple-tab-span" style="white-space: pre;"> </span>(ocf::heartbeat:IPaddr2):<span class="Apple-tab-span" style="white-space: pre;"> </span>Started sql01b<br><br><br>Starting the cluster tools on sql01a (rcopenais start) leads to the following lines (snipped the OK-looking lines at top as I enabled debugging in corosync.conf; if necessary i can of course post the whole output, but thats a lot of lines ...) in /var/log/messages:<br><br><span class="Apple-converted-space"> </span><br>[snip]<br>Jun 17 18:20:18 sql01a corosync[14051]: [MAIN ] main.c:278 Completed service synchronization, ready to provide service.<br>Jun 17 18:20:18 sql01a corosync[14051]: [TOTEM ] totempg.c:292 waiting_trans_ack changed to 0<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a mgmtd: [14062]: debug: main: run the loop...<br>Jun 17 18:20:18 sql01a mgmtd: [14062]: info: Started.<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14056, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process cib no longer wishes to be respawned<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] utils.c:340 info: update_member: Node sql01a now has process list: 00000000000000000000000000151212 (1380882)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process crmd exited (pid=14061, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process crmd no longer wishes to be respawned<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] utils.c:340 info: update_member: Node sql01a now has process list: 00000000000000000000000000151012 (1380370)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process attrd exited (pid=14059, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process attrd no longer wishes to be respawned<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] utils.c:340 info: update_member: Node sql01a now has process list: 00000000000000000000000000150012 (1376274)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1672 debug: send_cluster_id: Local update: id=17499146, born=1452, seq=1452<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:486 ERROR: pcmk_wait_dispatch: Child process pengine exited (pid=14060, rc=100)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:490 notice: pcmk_wait_dispatch: Child process pengine no longer wishes to be respawned<br>[snip]<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1365 info: send_member_notification: Sending membership update 1452 to 1 children<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:962 debug: pcmk_cluster_id_callback: Node update: sql01a (1.1.10)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:962 debug: pcmk_cluster_id_callback: Node update: sql01a (1.1.10)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:18 sql01a corosync[14051]: [pcmk ] plugin.c:1565 WARN: route_ais_message: Sending message to local.attrd failed: ipc delivery failed (rc=-2)<br>Jun 17 18:20:22 sql01a sshd[14068]: Connection closed by 10.4.1.60 [preauth]<br>Jun 17 18:20:23 sql01a corosync[14051]: [CONFDB] confdb.c:405 lib_init_fn: conn=0x663570<br>Jun 17 18:20:23 sql01a corosync[14051]: [CONFDB] confdb.c:411 exit_fn for conn=0x663570<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: error: setup_cib: Could not connect to the CIB service: Transport endpoint is not connected (-107)<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: crm_update_peer_state: plugin_handle_membership: Node sql01a[17499146] - state is now member (was (null))<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: get_node_name: Could not obtain a node name for classic openais (with plugin) nodeid 34276362<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: crm_update_peer_state: plugin_handle_membership: Node (null)[34276362] - state is now member (was (null))<br>Jun 17 18:20:27 sql01a stonith-ng[14057]: notice: plugin_handle_membership: Membership 1452: quorum acquired<br>[snip]<br><br><br>some more info:<br><br>sql01a:~ # corosync-cfgtool -s<br>Printing ring status.<br>Local node ID 17499146<br>RING ID 0<br><span class="Apple-tab-span" style="white-space: pre;"> </span>id<span class="Apple-tab-span" style="white-space: pre;"> </span>= 10.4.11.1<br><span class="Apple-tab-span" style="white-space: pre;"> </span>status<span class="Apple-tab-span" style="white-space: pre;"> </span>= ring 0 active with no faults<br>RING ID 1<br><span class="Apple-tab-span" style="white-space: pre;"> </span>id<span class="Apple-tab-span" style="white-space: pre;"> </span>= 10.255.4.1<br><span class="Apple-tab-span" style="white-space: pre;"> </span>status<span class="Apple-tab-span" style="white-space: pre;"> </span>= ring 1 active with no faults<br>sql01a:~ # corosync-objctl | grep member<br>runtime.totem.pg.mrp.srp.members.17499146.ip=r(0) ip(10.4.11.1) r(1) ip(10.255.4.1)<span class="Apple-converted-space"> </span><br>runtime.totem.pg.mrp.srp.members.17499146.join_count=1<br>runtime.totem.pg.mrp.srp.members.17499146.status=joined<br>runtime.totem.pg.mrp.srp.members.34276362.ip=r(0) ip(10.4.11.2) r(1) ip(10.255.4.2)<span class="Apple-converted-space"> </span><br>runtime.totem.pg.mrp.srp.members.34276362.join_count=1<br>runtime.totem.pg.mrp.srp.members.34276362.status=joined<br><br>This is similar to the other (working) host.<br><br>Interesting is:<br><br>sql01a:~ # rcopenais status<br>Running<br>sql01a:~ # ps ax | grep -E "pace|heart|coro" | grep -v grep<br>14051 ? Ssl 0:00 /usr/sbin/corosync<br>14057 ? S< 0:00 /usr/lib64/pacemaker/stonithd<br>14058 ? S< 0:00 /usr/lib64/pacemaker/lrmd<br>14062 ? S< 0:00 /usr/lib64/heartbeat/mgmtd<br>sql01a:~ #<span class="Apple-converted-space"> </span><br><br><br><br>For digging further whats going wrong I then started the missing processes (cib, attrd, pengine and crmd) from the command line (using the -V switch). cib, attrd and pengine are starting fine (even crm now shows a result, although wrong) but crmd fails:<br><br><br>sql01a:~ # /usr/lib64/pacemaker/crmd -V<br>notice: main:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>CRM Git Version: f3eeaf4<br> info: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_STARTUP from crmd_init() received in state S_STARTING<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_clm' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_evt' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_ckpt' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_amf_v2' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_msg' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_lck' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'openais_tmr' for option: name<br> info: config_find_next:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Processing additional service options...<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found 'pacemaker' for option: name<br> info: get_config_opt:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Found '0' for option: ver<br> info: get_cluster_type:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Detected an active 'classic openais (with plugin)' cluster<br> info: do_cib_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>CIB connection established<br>notice: crm_cluster_connect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connecting to cluster infrastructure: classic openais (with plugin)<br> info: init_cs_connection_classic:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Creating connection to our Corosync plugin<br> info: init_cs_connection_classic:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>AIS connection established<br>notice: get_node_name:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Defaulting to uname -n for the local classic openais (with plugin) node name<br> info: plugin_get_details:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Server details: id=17499146 uname=sql01a cname=pcmk<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Created entry 6002e998-b8ba-4daa-9c5d-1082bd1ab73a/0x65b8a0 for node sql01a/17499146 (1 total)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 17499146 is now known as sql01a<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01a is now (null)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 17499146 has uuid sql01a<br> info: crm_update_peer_proc:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>init_cs_connection_classic: Node sql01a[17499146] - unknown is now online<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Client sql01a/peer now has status [online] (DC=<null>)<br> info: init_cs_connection_once:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connection to 'classic openais (with plugin)': established<br>notice: get_node_name:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Defaulting to uname -n for the local classic openais (with plugin) node name<br> info: do_ha_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connected to the cluster<br> info: lrmd_ipc_connect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connecting to lrmd<br> info: do_lrm_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>LRM connection established<br> info: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Delaying start, no membership data (0000000000100000)<br> info: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Delaying start, no membership data (0000000000100000)<br> info: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Delaying start, no membership data (0000000000100000)<br>notice: plugin_handle_membership:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Membership 1460: quorum acquired<br> info: crm_update_peer_proc:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01a[17499146] - unknown is now member<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Client sql01a/peer now has status [offline] (DC=<null>)<br>notice: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Our peer connection failed<br>notice: crm_update_peer_state:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01a[17499146] - state is now member (was (null))<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01a is now member (was (null))<br> info: crm_update_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01a: id=17499146 state=member addr=r(0) ip(10.4.11.1) r(1) ip(10.255.4.1) (new) votes=1 (new) born=1460 seen=1460 proc=00000000000000000000000000140012<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Created entry 8925fb86-2b70-4c91-af03-a8251a9182fc/0x65ecf0 for node sql01b/34276362 (2 total)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 34276362 is now known as sql01b<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01b is now (null)<br> info: crm_get_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Node 34276362 has uuid sql01b<br> info: crm_update_peer_proc:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01b[34276362] - unknown is now member<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Client sql01b/peer now has status [online] (DC=<null>)<br>notice: crm_update_peer_state:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01b[34276362] - state is now member (was (null))<br> info: peer_update_callback:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>sql01b is now member (was (null))<br> info: crm_update_peer:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>plugin_handle_membership: Node sql01b: id=34276362 state=member addr=r(0) ip(10.4.11.2) r(1) ip(10.255.4.2) (new) votes=1 (new) born=1436 seen=1460 proc=00000000000000000000000000151312<br> error: reap_dead_nodes:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>We're not part of the cluster anymore<br> info: plugin_handle_membership:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Membership 1460: quorum retained<br> error: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_ERROR from peer_update_callback() received in state S_STARTING<br>notice: do_state_transition:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>State transition S_STARTING -> S_RECOVERY [ input=I_ERROR cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]<br>warning: do_recover:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Fast-tracking shutdown in response to errors<br> error: do_started:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Start cancelled... S_RECOVERY<br> error: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_ERROR from reap_dead_nodes() received in state S_RECOVERY<br> info: do_dc_release:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>DC role released<br> info: do_te_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Transitioner is now inactive<br> error: do_log:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY<br> info: do_state_transition:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ]<br>notice: lrm_state_verify_stopped:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Stopped 0 recurring operations at shutdown (0 ops remaining)<br> info: do_lrm_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from the LRM<br> info: lrmd_api_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from lrmd service<br> info: lrmd_ipc_connection_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>IPC connection destroyed<br> info: lrm_connection_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>LRM Connection disconnected<br> info: lrmd_api_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from lrmd service<br>notice: do_lrm_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnected from the LRM<br> info: crm_cluster_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from cluster infrastructure: classic openais (with plugin)<br>notice: terminate_cs_connection:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting from Corosync<br> info: cluster_disconnect_cpg:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>No CPG connection<br> info: crm_cluster_disconnect:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnected from classic openais (with plugin)<br> info: do_ha_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnected from the cluster<br> info: do_cib_control:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Disconnecting CIB<br> info: crmd_cib_connection_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Connection to the CIB terminated...<br> info: do_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Performing A_EXIT_0 - gracefully exiting the CRMd<br> info: do_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>[crmd] stopped (0)<br> info: crmd_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Dropping I_RELEASE_SUCCESS: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_dc_release ]<br> info: crmd_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ]<br> info: crmd_cs_destroy:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>connection closed<br> info: crmd_init:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>13510 stopped: OK (0)<br> error: crmd_fast_exit:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Could not recover from internal error<br> info: crm_xml_cleanup:<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Cleaning up memory from libxml2<br>sql01a:~ #<span class="Apple-converted-space"> </span><br><br><br>what I tried up to now:<br>- stop sql01a, remove all cib.xml and crm.xml in their folders and restart sql01a (I know this is quite harsh but what did I have to loose
) -> the config on sql01a has been taken from sql01b but the crmd always dies as before (same error messages)<br>- I removed all cluster related packages (pacemaker, corosync, drbd,
) from sql01a and deleted the remaining dirs and config files, reinstalled all with copies from sql01b -> same result.<br>- I removed the whole crm config from sql01b (the working cluster host) using cibadmin -Q force and re-added the config (using crm commands") -> same result<br>- of course rebooted/restarted both servers several times -> nothing changes
<br><br><br>As deleting the config from sql01a (several times) I thought the error must be in the config/communication on/with sql01b but I also removed and added the config there, in addition there is now firewall between them and there is communication.<br><br>Does anybody know/understand what is going on here?<br>How can I solve this issue and get my cluster back to work as it did for quite a long time recently?<br><br><br>thanks in advance for any hints!<br><br>best regards,<br> Markus<br><br><br><br><br>-- .- .-. -.- ..- ... / -.- .-. .- ..- ... .<br>Markus Krause<br>Max Planck Institute of Biochemistry<br>- Computing Center (core services, networking, OS X)<br>Am Klopferspitz 18<br>D-82152 Martinsried, Germany<br>Phone: +49(89)8578-2825<br><br>_______________________________________________<br>Pacemaker mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><br>Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a><br></blockquote><br>_______________________________________________<br>Pacemaker mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><br>Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a></div></blockquote></div><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div apple-content-edited="true" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px;"><font class="Apple-style-span" color="#033EFC">-- .- .-. -.- ..- ... / -.- .-. .- ..- ... .<br>Markus Krause<br>Max Planck Institute of Biochemistry<br>- Computing Center (core services, networking, OS X)<br>Am Klopferspitz 18<br>D-82152 Martinsried, Germany<br>Phone: +49(89)8578-2825</font></span></font></div></div></div><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Pacemaker mailing list:<span class="Apple-converted-space"> </span></span><a href="mailto:Pacemaker@oss.clusterlabs.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Pacemaker@oss.clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Project Home:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://www.clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Getting started:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Bugs:<span class="Apple-converted-space"> </span></span><a href="http://bugs.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://bugs.clusterlabs.org</a></blockquote></div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" size="2"><span class="Apple-style-span" style="font-size: 10px; "><font class="Apple-style-span" color="#033EFC">-- .- .-. -.- ..- ... / -.- .-. .- ..- ... .<br>Markus Krause<br>Max Planck Institute of Biochemistry<br>- Computing Center (core services, networking, OS X)<br>Am Klopferspitz 18<br>D-82152 Martinsried, Germany<br>Phone: +49(89)8578-2825</font></span></font></div></div>
</div>
<br></body></html>