No subject
Mon Mar 29 07:57:18 UTC 2010
(exit 6), which means "program is not configured". <br>
<pre wrap="">Jul 7 11:47:58 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster</pre>
Because the mysql-server primitive and/or the mysql RA were not
properly configured at the time the error message was written, it led
to:<br>
<br>
<pre wrap="">Jul 7 11:47:58 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)</pre>
<br>
Therefore, export the variables manually, check the exit codes on each
operation of the script, they should match normal operation as
described in the RA specification draft, if everything is OK, move on
the the crm shell, cleanup the mysql-server resource, cleanup the max
failures (the cluster resource manager keeps track of these, I've had
to manually remove everything from /var/lib/heartbeat/crm/*, haven't
found a way to clear this option, yet, and loaded the saved config with
crm load config.crm to have the cluster functional again). <br>
<br>
Also keep in mind that the mysql RA script needs to be the same (as
well as /etc/my.cnf) on all cluster nodes, and even if Pacemaker
propagates the changes to all nodes via multicast,
/var/lib/heartbeat/crm/* files are placed on all cluster nodes as well.
And the mysql RA script needs to work the same way on all nodes.<br>
<br>
And I'll stop, I've said enough already.<br>
<br>
Cheers.<br>
<span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; font-size: medium;"></span><br>
<br>
Jake Bogie wrote:
<blockquote
cite="mid:4E361586C068AC4FB5CEB8C37BA0232A2325FC at SSMSX01.Mythos.SureSource.com"
type="cite">
<pre wrap="">So I took Dan's advice this time and cleaned up my resource
configuration, updated the script, and verified...however I'm still not
getting the resource online...
[root at qad01 heartbeat]# crm resource start mysql-server
[root at qad01 heartbeat]# crm status
============
Last updated: Wed Jul 7 11:49:20 2010
Stack: openais
Current DC: qad01 - partition with quorum
Version: 1.0.9-89bd754939df5150de7cd76835f98fe90851b677
2 Nodes configured, 2 expected votes
3 Resources configured.
============
Online: [ qad02 qad01 ]
Resource Group: mysql
fs_mysql (ocf::heartbeat:Filesystem): Started qad01
ip_mysql (ocf::heartbeat:IPaddr2): Started qad01
Master/Slave Set: ms_drbd_mysql
Masters: [ qad01 ]
Slaves: [ qad02 ]
Failed actions:
mysql-server_start_0 (node=qad01, call=6, rc=6, status=complete):
not configured
[root at qad01 heartbeat]# ./mysql start
mysql[5750]: DEBUG: MySQL is not running
mysql[5750]: DEBUG: MySQL is not running
100707 11:49:55 [Warning] option 'group_concat_max_len': unsigned value
0 adjusted to 4
100707 11:49:55 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
100707 11:49:55 InnoDB: highest supported file format is Barracuda.
100707 11:49:55 InnoDB: Warning: allocated tablespace 1, old maximum
was 0
100707 11:49:55 InnoDB Plugin 1.0.9 started; log sequence number
28732335894
100707 11:49:55 [Note] Event Scheduler: Loaded 0 events
100707 11:49:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.48-community' socket: '/var/lib/mysql/mysql.sock' port:
3306 MySQL Community Server (GPL)
mysql[5750]: INFO: MySQL started
[root at qad01 heartbeat]# ./mysql status
[root at qad01 heartbeat]# ./mysql monitor
[root at qad01 heartbeat]# ./mysql validate-all
[root at qad01 heartbeat]# crm status
============
Last updated: Wed Jul 7 11:50:23 2010
Stack: openais
Current DC: qad01 - partition with quorum
Version: 1.0.9-89bd754939df5150de7cd76835f98fe90851b677
2 Nodes configured, 2 expected votes
3 Resources configured.
============
Online: [ qad02 qad01 ]
Resource Group: mysql
fs_mysql (ocf::heartbeat:Filesystem): Started qad01
ip_mysql (ocf::heartbeat:IPaddr2): Started qad01
Master/Slave Set: ms_drbd_mysql
Masters: [ qad01 ]
Slaves: [ qad02 ]
Failed actions:
mysql-server_start_0 (node=qad01, call=6, rc=6, status=complete):
not configured
[root at qad01 heartbeat]# ./mysql stop
100707 11:50:31 [Note] /usr/sbin/mysqld: Normal shutdown
./mysql: line 426: (/1000)-5: syntax error: operand expected (error
token is "/1000)-5")
100707 11:50:31 [Note] Event Scheduler: Purging the queue. 0 events
100707 11:50:31 InnoDB: Starting shutdown...
[root at qad01 heartbeat]# 100707 11:50:36 InnoDB: Shutdown completed; log
sequence number 28732335904
100707 11:50:36 [Note] /usr/sbin/mysqld: Shutdown complete
[root at qad01 heartbeat]#
[root at qad01 heartbeat]# crm configure show mysql-server
primitive mysql-server ocf:heartbeat:mysql \
op monitor interval="30s" timeout="30s" \
op start interval="0" timeout="120" \
op stop interval="0" timeout="120" \
params binary="/usr/sbin/mysqld" config="/etc/my.cnf"
datadir="/drbd/mysql/data/" user="mysql" group="mysql"
log="/var/log/mysqld.log" pid="/drbd/mysql/data/qadb.pid"
socket="/var/lib/mysql/mysql.sock" test_user="qaclus"
test_passwd="isitup" test_table="cluster_check.connectioncheck" \
meta target-role="Started"
[root at qad01 heartbeat]# cat mysql
#!/bin/sh
#
#
# MySQL
#
# Description: Manages a MySQL database as Linux-HA resource
#
# Author: Alan Robertson : DB2 Script
# Author: Jakub Janczak : Rewrite as MySQL
# Author: Andrew Beekhof : Cleanup and import
# Author: Sebastian Reitenbach : add OpenBSD defaults, more
cleanup
# Author: Narayan Newton : Add Gentoo/Debian defaults
#
# Support: <a class="moz-txt-link-abbreviated" href="mailto:linux-ha at lists.linux-ha.org">linux-ha at lists.linux-ha.org</a>
# License: GNU General Public License (GPL)
# Copyright: (C) 2002 - 2005 International Business Machines, Inc.
#
# An example usage in /etc/ha.d/haresources:
# node1 10.0.0.170 mysql
#
# See usage() function below for more details...
#
# OCF instance parameters:
# OCF_RESKEY_binary
# OCF_RESKEY_config
# OCF_RESKEY_datadir
# OCF_RESKEY_user
# OCF_RESKEY_group
# OCF_RESKEY_test_table
# OCF_RESKEY_test_user
# OCF_RESKEY_test_passwd
# OCF_RESKEY_enable_creation
# OCF_RESKEY_additional_parameters
# OCF_RESKEY_log
# OCF_RESKEY_pid
# OCF_RESKEY_socket
#######################################################################
# Initialization:
: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/resource.d/heartbeat}
. ${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs
#######################################################################
# Added exports
export OCF_ROOT=/usr/lib/ocf/
export OCF_RESKEY_binary_default="/usr/sbin/mysqld"
export OCF_RESKEY_config_default="/etc/my.cnf"
export OCF_RESKEY_datadir_default="/drbd/mysql/data"
export OCF_RESKEY_user_default="mysql"
export OCF_RESKEY_group_default="mysql"
export OCF_RESKEY_log_default="/var/log/mysqld.log"
export OCF_RESKEY_pid_default="/drbd/mysql/data/qadb.pid"
export OCF_RESKEY_socket_default="/var/lib/mysql/mysql.sock"
export OCF_RESKEY_test_user_default="qaclus"
export OCF_RESKEY_test_table_default="cluster_check.connectioncheck"
export OCF_RESKEY_test_passwd_default="isitup"
# Fill in some defaults if no values are specified
HOSTOS=`uname`
if [ "X${HOSTOS}" = "XOpenBSD" ];then
OCF_RESKEY_binary_default="/usr/local/bin/mysqld_safe"
OCF_RESKEY_config_default="/etc/my.cnf"
OCF_RESKEY_datadir_default="/var/mysql"
OCF_RESKEY_user_default="_mysql"
OCF_RESKEY_group_default="_mysql"
OCF_RESKEY_log_default="/var/log/mysqld.log"
OCF_RESKEY_pid_default="/var/mysql/mysqld.pid"
OCF_RESKEY_socket_default="/var/run/mysql/mysql.sock"
OCF_RESKEY_test_user_default="root"
OCF_RESKEY_test_table_default="mysql.user"
OCF_RESKEY_test_passwd_default=""
OCF_RESKEY_enable_creation_default=0
OCF_RESKEY_additional_parameters_default=""
else
OCF_RESKEY_binary_default="/usr/sbin/mysqld"
OCF_RESKEY_config_default="/etc/my.cnf"
OCF_RESKEY_datadir_default="/drbd/mysql/data"
OCF_RESKEY_user_default="mysql"
OCF_RESKEY_group_default="mysql"
OCF_RESKEY_log_default="/var/log/mysqld.log"
OCF_RESKEY_pid_default="/drbd/mysql/data/qadb.pid"
OCF_RESKEY_socket_default="/var/lib/mysql/mysql.sock"
OCF_RESKEY_test_user_default="qaclus"
OCF_RESKEY_test_table_default="cluster_check.connectioncheck"
OCF_RESKEY_test_passwd_default="isitup"
OCF_RESKEY_enable_creation_default=0
OCF_RESKEY_additional_parameters_default=""
Fi
[root at qad01 heartbeat]# cat /var/log/messages | grep mysql-server
Jul 7 11:43:38 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:43:38 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:43:38 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:43:38 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:43:38 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:43:38 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:43:38 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
Jul 7 11:47:58 qad01 crmd: [4360]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=mysql-server_monitor_0,
magic=0:7;7:0:7:e87a73c4-97b8-4f63-9e69-89ec59fce708, cib=0.287.3) :
Resource op removal
Jul 7 11:47:58 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:47:58 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:47:58 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:47:58 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:47:58 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:47:58 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:47:58 qad01 attrd: [4358]: info: attrd_trigger_update: Sending
flush op to all hosts for: fail-count-mysql-server (INFINITY)
Jul 7 11:47:58 qad01 attrd: [4358]: info: attrd_trigger_update: Sending
flush op to all hosts for: last-failure-mysql-server (1278516515)
Jul 7 11:47:58 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
Jul 7 11:47:58 qad01 crmd: [4360]: info: te_rsc_command: Initiating
action 7: monitor mysql-server_monitor_0 on qad02
Jul 7 11:47:58 qad01 crmd: [4360]: info: match_graph_event: Action
mysql-server_monitor_0 (7) confirmed on qad02 (rc=0)
Jul 7 11:47:58 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:47:58 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:47:58 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:47:58 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:47:58 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:47:58 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:47:58 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
Jul 7 11:48:01 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:48:01 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:48:01 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:48:01 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:48:01 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:48:01 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:48:01 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
Jul 7 11:48:10 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:48:10 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:48:10 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:48:10 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:48:10 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:48:10 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:48:10 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
Jul 7 11:48:11 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:48:11 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:48:11 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:48:11 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:48:11 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:48:11 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:48:11 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
Jul 7 11:48:26 qad01 pengine: [4359]: ERROR: unpack_rsc_op: Hard error
- mysql-server_start_0 failed with rc=6: Preventing mysql-server from
re-starting anywhere in the cluster
Jul 7 11:48:26 qad01 pengine: [4359]: WARN: unpack_rsc_op: Processing
failed op mysql-server_start_0 on qad01: not configured (6)
Jul 7 11:48:26 qad01 pengine: [4359]: notice: native_print:
mysql-server (ocf::heartbeat:mysql): Stopped
Jul 7 11:48:26 qad01 pengine: [4359]: info: get_failcount: mysql-server
has failed INFINITY times on qad01
Jul 7 11:48:26 qad01 pengine: [4359]: WARN: common_apply_stickiness:
Forcing mysql-server away from qad01 after 1000000 failures
(max=1000000)
Jul 7 11:48:26 qad01 pengine: [4359]: info: native_color: Resource
mysql-server cannot run anywhere
Jul 7 11:48:26 qad01 pengine: [4359]: notice: LogActions: Leave
resource mysql-server (Stopped)
___
Message: 7
Date: Wed, 07 Jul 2010 12:55:51 +0300
From: Dan Frincu <a class="moz-txt-link-rfc2396E" href="mailto:dfrincu at streamwide.ro"><dfrincu at streamwide.ro></a>
To: The Pacemaker cluster resource manager
<a class="moz-txt-link-rfc2396E" href="mailto:pacemaker at oss.clusterlabs.org"><pacemaker at oss.clusterlabs.org></a>
Subject: Re: [Pacemaker] Upgraded mysql from 5.0 to 5.1
Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:4C344F27.1060707 at streamwide.ro"><4C344F27.1060707 at streamwide.ro></a>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Have you copied twice this line?
socket="/var/lib/mysql/mysql.sock" binary="/usr/sbin/mysqld"
socket="/var/lib/mysql/mysql.sock" binary="/usr/sbin/mysqld"
I think so. Regardless, to test a resource agent manually requires that
you define some variables and then call the script by hand. Also, check
all the actions (start,stop,restart,promote,etc) and their exit codes,
to see if they match the OCF RA specification. Most of the problems that
you will have with a resource agent and it's resource can be found if
you're manually testing the RA script.
Go to /usr/lib/ocf/resource.d/heartbeat/
Open the mysql RA script. Go to line 63 and starting from that line
update the values in the script to match the contents of /etc/my.cnf.
Then update the crm configure for the primitive mysql-server to match as
well.
From what I remember, the values in
OCF_RESKEY_{binary_default,pid_default,socket_default} are wrong in the
RA script vs what's actually installed.
Then "export OCF_ROOT=/usr/lib/ocf/" and all OCF_RESKEY_* with their
defined values, then call the script with no parameters. It should
provide the usage of the script. Then take step by step each action and
check it's exit code, see if it matches the OCF RA specification, and
also check to see if it actually starts the resource or not. The thing
is, once the script works as it should, all the issues have been
resolved, the cluster will work with the mysql-server resource.
Regards,
Dan
Jake Bogie wrote:
</pre>
<blockquote type="cite">
<pre wrap="">So I took Raoul's advice and ditched the lsb:mysql check and went for
the ocf:heartbeat version however...
I'm getting this now...
What am I missing? I'm having a hard time finding a document on how to
setup this resource agent.
============
Last updated: Tue Jul 6 12:44:07 2010
Stack: openais
Current DC: qad02 - partition with quorum
Version: 1.0.9-89bd754939df5150de7cd76835f98fe90851b677
2 Nodes configured, 2 expected votes
3 Resources configured.
============
Online: [ qad02 qad01 ]
Resource Group: mysql
fs_mysql (ocf::heartbeat:Filesystem): Started qad02
ip_mysql (ocf::heartbeat:IPaddr2): Started qad02
Master/Slave Set: ms_drbd_mysql
Masters: [ qad02 ]
Slaves: [ qad01 ]
Failed actions:
mysql-server_start_0 (node=qad01, call=6, rc=6, status=complete):
not configured
mysql-server_start_0 (node=qad02, call=33, rc=5, status=complete):
not installed
###
primitive mysql-server ocf:heartbeat:mysql \
op monitor interval="30s" timeout="30s" \
op start interval="0" timeout="120" \
op stop interval="0" timeout="120" \
params config="/etc/my.cnf" datadir="/drbd/mysql/data/"
socket="/var/lib/mysql/mysql.sock" binary="/usr/sbin/mysqld"
socket="/var/lib/mysql/mysql.sock" binary="/usr/sbin/mysqld"
pid="/drbd/mysql/data/mysql.pid" test_passwd="isitup"
test_table="cluster_check.connectioncheck" test_user="qaclus" \
_______________________________________________
Pacemaker mailing list: <a class="moz-txt-link-abbreviated" href="mailto:Pacemaker at oss.clusterlabs.org">Pacemaker at oss.clusterlabs.org</a>
<a class="moz-txt-link-freetext" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a>
Project Home: <a class="moz-txt-link-freetext" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a>
Getting started:
</pre>
</blockquote>
<pre wrap=""><!----><a class="moz-txt-link-freetext" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
</pre>
<blockquote type="cite">
<pre wrap="">Bugs:
</pre>
</blockquote>
<pre wrap=""><!----><a class="moz-txt-link-freetext" href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemake">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemake</a>
r
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania
E-mail: <a class="moz-txt-link-abbreviated" href="mailto:dfrincu at streamwide.ro">dfrincu at streamwide.ro</a>
Phone: +40 (0) 21 320 41 24
</pre>
</body>
</html>
--------------050806000701030603050508--
More information about the Pacemaker
mailing list