[Pacemaker] pacemaker won't start mysql in the second node
Bart Coninckx
bart.coninckx at telenet.be
Tue Feb 1 20:17:53 UTC 2011
Ha, hadn't heard of that book yet. Too bad it's in German. Mein Deutsch
is nicht so slecht aber nicht gut genug für technische Zagen ;-)
On 02/01/11 21:00, paul harford wrote:
> Hi bart
> i think your right linit is looking the best or Michael Schwartzkopff's
> book if its avaiable in english
>
> Paul
>
> On 1 February 2011 19:44, Bart Coninckx <bart.coninckx at telenet.be
> <mailto:bart.coninckx at telenet.be>> wrote:
>
> Novell will not support you unless you use SLES, same for Redhat and
> RHEL. Not sure about IBM, but they are $$$. So the wonderful people of
> Linbit seem a good choice.
>
>
>
>
> On 02/01/11 18:20, Liang.Ma at asc-csa.gc.ca
> <mailto:Liang.Ma at asc-csa.gc.ca> wrote:
> > Hi There,
> >
> > I think we should seek professional help. Since the clusters we
> are going to build contain components such as heartbeat, corosync,
> pacemaker, drbd and ocfs2 or gfs2. And also I know there are quite a
> few companies who provide the support. So anyone has any suggestion,
> IBM, Redhat, Novell, or linbit?
> >
> > Thanks in advance.
> >
> > Liang Ma
> > Contractuel | Consultant | SED Systems Inc.
> > Ground Systems Analyst
> > Agence spatiale canadienne | Canadian Space Agency
> > 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9
> > Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083
> > Courriel/E-mail : [liang.ma <http://liang.ma>@space.gc.ca
> <http://space.gc.ca>]
> > Site web/Web site : [www.space.gc.ca <http://www.space.gc.ca> ]
> >
> >
> >
> >
> > -----Original Message-----
> > From: Ma, Liang
> > Sent: January 31, 2011 10:48 AM
> > To: 'The Pacemaker cluster resource manager'
> > Subject: RE: [Pacemaker] pacemaker won't start mysql in the second
> node
> >
> > Sorry for sending this again. The log format was a bit messy in
> the previous one.
> >
> > Thanks for your hints. I went through the cluster logs more
> carefully. By comparing the logs from the two nodes, the real
> different is after the line
> >
> > info: process_lrm_event: LRM operation fs_mysql_start_0
> >
> > On node arsvr1, after that line we got a confirmation on Action
> fs_mysql_start_0 as such
> >
> > info: match_graph_event: Action fs_mysql_start_0 (8) confirmed on
> arsvr1
> >
> > and then went on to Initiating action 9: start mysql_start_0 on
> arsvr1 (local).
> >
> > However on node arsvr2, we never see the confirmation from Action
> fs_mysql_start_0. So mysql_start_0 is never called. But the strange
> thing is, I can see the drbd partition of fs_mysql is properly
> mounted on arsvr2. Anyone knows what might stop arsvr2 to run that
> Action fs_mysql_start_0 (8) confirmed?
> >
> > Thanks in advance.
> >
> > Here are the logs from the two nodes.
> >
> > Logs on Node 2:
> >
> > Jan 28 14:24:23 arsvr2 lrmd: [919]: info: rsc:fs_mysql:229: start
> > Jan 28 14:24:23 arsvr2 Filesystem[1568]: [1596]: INFO: Running
> start for /dev/drbd/by-res/r0 on /var/lib/mysql
> > Jan 28 14:24:23 arsvr2 lrmd: [919]: info: RA output:
> (fs_mysql:start:stderr) FATAL: Module scsi_hostadapter not found.
> > Jan 28 14:24:23 arsvr2 Filesystem[1568]: [1606]: INFO: Starting
> filesystem check on /dev/drbd/by-res/r0
> > Jan 28 14:24:23 arsvr2 lrmd: [919]: info: RA output:
> (fs_mysql:start:stdout) fsck from util-linux-ng 2.17.2
> > Jan 28 14:24:23 arsvr2 lrmd: [919]: info: RA
> output:(fs_mysql:start:stdout) /dev/drbd0: clean, 178/3276800 files,
> 257999/13106791 blocks
> > Jan 28 14:24:23 arsvr2 crmd: [922]: info: process_lrm_event: LRM
> operation fs_mysql_start_0 (call=229, rc=0,
> cib-update=251,confirmed=true) ok
> > Jan 28 14:24:46 arsvr2 cib: [918]: info: cib_stats: Processed 149
> operations (0.00us average, 0% utilization) in the last 10min
> >
> > Logs on Node 1:
> >
> > Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: rsc:fs_mysql:867: start
> > Jan 28 14:28:58 arsvr1 crmd: [1068]: info: te_rsc_command:
> Initiating action 31: monitor drbd_mysql:1_monitor_15000 on arsvr2
> > Jan 28 14:28:58 arsvr1 Filesystem[516]: [544]: INFO: Running start
> for /dev/drbd/by-res/r0 on /var/lib/mysql
> > Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: RA
> output:(fs_mysql:start:stderr) FATAL: Module scsi_hostadapter not found.
> > Jan 28 14:28:58 arsvr1 Filesystem[516]: [554]: INFO: Starting
> filesystem check on /dev/drbd/by-res/r0
> > Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: RA
> output:(fs_mysql:start:stdout) fsck from util-linux-ng 2.17.2
> > Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: RA
> output:(fs_mysql:start:stdout) /dev/drbd0: clean, 178/3276800
> files,257999/13106791 blocks
> > Jan 28 14:28:58 arsvr1 crmd: [1068]: info: process_lrm_event: LRM
> operation fs_mysql_start_0 (call=867, rc=0,
> cib-update=1650,confirmed=true) ok
> > Jan 28 14:28:58 arsvr1 crmd: [1068]: info: match_graph_event:
> Action fs_mysql_start_0 (8) confirmed on arsvr1 (rc=0)
> > Jan 28 14:28:58 arsvr1 crmd: [1068]: info: te_rsc_command:
> Initiating action 9: start mysql_start_0 on arsvr1 (local)
> > Jan 28 14:28:58 arsvr1 crmd: [1068]: info: do_lrm_rsc_op:
> Performing key=9:551:0:9c402121-906c-42de-a18a-68deb24208cb
> op=mysql_start_0 )
> > Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: rsc:mysql:868: start
> > Jan 28 14:28:58 arsvr1 mysqld_safe: Starting mysqld daemon with
> databases from /var/lib/mysql
> > Jan 28 14:28:59 arsvr1 crmd: [1068]: info: match_graph_event:
> Action drbd_mysql:1_monitor_15000 (31) confirmed on arsvr2 (rc=0)
> > Jan 28 14:29:02 arsvr1 mysql[576]: [728]: INFO: MySQL started
> > Jan 28 14:29:02 arsvr1 crmd: [1068]: info: process_lrm_event: LRM
> operation mysql_start_0 (call=868, rc=0,
> cib-update=1651,confirmed=true) ok
> > Jan 28 14:29:02 arsvr1 crmd: [1068]: info: match_graph_event:
> Action mysql_start_0 (9) confirmed on arsvr1 (rc=0)
> >
> >
> > Liang Ma
> > Contractuel | Consultant | SED Systems Inc.
> > Ground Systems Analyst
> > Agence spatiale canadienne | Canadian Space Agency
> > 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9
> > Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083
> > Courriel/E-mail : [liang.ma <http://liang.ma>@space.gc.ca
> <http://space.gc.ca>]
> > Site web/Web site : [www.space.gc.ca <http://www.space.gc.ca> ]
> >
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Dejan Muhamedagic [mailto:dejanmm at fastmail.fm
> <mailto:dejanmm at fastmail.fm>]
> > Sent: January 28, 2011 11:09 AM
> > To: The Pacemaker cluster resource manager
> > Subject: Re: [Pacemaker] pacemaker won't start mysql in the second
> node
> >
> > On Fri, Jan 28, 2011 at 08:50:45AM -0500, Liang.Ma at asc-csa.gc.ca
> <mailto:Liang.Ma at asc-csa.gc.ca> wrote:
> >> Hi Dejan, thanks for your reply.
> >>
> >> That's one of the problem. I don't see any logs in log file
> /var/log/mysql/error.log.
> >
> > I meant the cluster logs.
> >
> >> I checked the permission of directories /var/run/mysqld and
> /var/log/mysql. In both nodes they are the same as
> >>
> >> drwxr-xr-x 2 mysql root 40 2011-01-27 13:50 /var/run/mysqld/
> >> drwxr-s--- 2 mysql adm 4096 2011-01-27 11:34 /var/log/mysql
> >>
> >> By the way, under which user pacemaker runs, root or someone else?
> >
> > pacemaker is a collection of programs. At any rate, the RA run
> > as root, but may su to another user (mysql) depending on the
> > resource configuration.
> >
> > Thanks,
> >
> > Dejan
> >
> >> Liang Ma
> >> Contractuel | Consultant | SED Systems Inc.
> >> Ground Systems Analyst
> >> Agence spatiale canadienne | Canadian Space Agency
> >> 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9
> >> Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083
> >> Courriel/E-mail : [liang.ma <http://liang.ma>@space.gc.ca
> <http://space.gc.ca>]
> >> Site web/Web site : [www.space.gc.ca <http://www.space.gc.ca> ]
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Dejan Muhamedagic [mailto:dejanmm at fastmail.fm
> <mailto:dejanmm at fastmail.fm>]
> >> Sent: January 28, 2011 8:26 AM
> >> To: The Pacemaker cluster resource manager
> >> Subject: Re: [Pacemaker] pacemaker won't start mysql in the
> second node
> >>
> >> Hi,
> >>
> >> On Thu, Jan 27, 2011 at 11:51:31AM -0500, Liang.Ma at asc-csa.gc.ca
> <mailto:Liang.Ma at asc-csa.gc.ca> wrote:
> >>>
> >>>
> >>> Hi There,
> >>>
> >>> I have set up a pair of ha LAMP servers using heartbeat,
> pacemaker and
> >>> drbd on Ubuntu 10.04 LTS. Everything works fine until I upgraded
> >>> mysql-server from 5.1.41-3ubuntu12.6 to 5.1.41-3ubuntu12.9. Now
> node 1
> >>> (arsvr1) works still fine, but mysql on node 2 (arsvr2) won't start
> >>> when I switch arsvr1 standby. The error message shown from "crm
> >>> status" is
> >>>
> >>> Failed actions:
> >>> mysql_start_0 (node=arsvr2, call=32, rc=4, status=complete):
> >>> insufficient privileges
> >>>
> >>> No errors logged in /var/log/mysql/error.log at all.
> >>
> >> I think that you should check directory permissions. The log
> >> file should give you a hint.
> >>
> >> Thanks,
> >>
> >> Dejan
> >>
> >>
> >>> drbd mysql partition mounted properly. If I go to
> >>> /usr/lib/ocf/resource.d/heartbeat and set the OCF_RESKEY
> parameters, I
> >>> have no problem to start mysql server by "./mysql start". But the
> >>> resource mysql won't show up in crm status.
> >>>
> >>> So looks somehow pacemaker fail to start resource mysql even before
> >>> running the resource script.
> >>>
> >>> Here is the configuration
> >>>
> >>> node $id="bc6bf61d-6b5f-4307-85f3-bf7bb11531bb" arsvr2 \
> >>> attributes standby="off"
> >>> node $id="bf0e7394-9684-42b9-893b-5a9a6ecddd7e" arsvr1 \
> >>> attributes standby="off"
> >>> primitive apache2 lsb:apache2 \
> >>> op start interval="0" timeout="60" \
> >>> op stop interval="0" timeout="120" start-delay="15" \
> >>> meta target-role="Started"
> >>> primitive drbd_mysql ocf:linbit:drbd \
> >>> params drbd_resource="r0" \
> >>> op monitor interval="15s"
> >>> primitive drbd_webfs ocf:linbit:drbd \
> >>> params drbd_resource="r1" \
> >>> op monitor interval="15s" \
> >>> op start interval="0" timeout="240" \
> >>> op stop interval="0" timeout="100"
> >>> primitive fs_mysql ocf:heartbeat:Filesystem \
> >>> params device="/dev/drbd/by-res/r0"
> directory="/var/lib/mysql" fstype="ext4" \
> >>> op start interval="0" timeout="60" \
> >>> op stop interval="0" timeout="120" \
> >>> meta target-role="Started"
> >>> primitive fs_webfs ocf:heartbeat:Filesystem \
> >>> params device="/dev/drbd/by-res/r1" directory="/srv"
> fstype="ext4" \
> >>> op start interval="0" timeout="60" \
> >>> op stop interval="0" timeout="120" \
> >>> meta target-role="Started"
> >>> primitive ip1 ocf:heartbeat:IPaddr2 \
> >>> params ip="10.10.10.193" nic="eth0" \
> >>> op monitor interval="5s"
> >>> primitive ip1arp ocf:heartbeat:SendArp \
> >>> params ip="10.10.10.193" nic="eth0"
> >>> primitive mysql ocf:heartbeat:mysql \
> >>> params binary="/usr/bin/mysqld_safe" config="/etc/mysql/my.cnf"
> >>> user="mysql" group="mysql" log="/var/log/mysql.log"
> >>> pid="/var/run/mysqld/mysqld.pid" datadir="/var/lib/mysql"
> >>> socket="/var/run/mysqld/mysqld.sock" \
> >>> op monitor interval="30s" timeout="30s" \
> >>> op start interval="0" timeout="120" \
> >>> op stop interval="0" timeout="120" \
> >>> meta target-role="Started"
> >>> group MySQLDB fs_mysql mysql \
> >>> meta target-role="Started"
> >>> group WebServices ip1 ip1arp fs_webfs apache2 \
> >>> meta target-role="Started"
> >>> ms ms_drbd_mysql drbd_mysql \
> >>> meta master-max="1" master-node-max="1" clone-max="2"
> >>> clone-node-max="1" notify="true"
> >>> ms ms_drbd_webfs drbd_webfs \
> >>> meta master-max="1" master-node-max="1" clone-max="2"
> >>> clone-node-max="1" notify="true" target-role="Started"
> >>> colocation apache2_with_ip inf: apache2 ip1
> >>> colocation apache2_with_mysql inf: apache2 ms_drbd_mysql:Master
> >>> colocation apache2_with_webfs inf: apache2 ms_drbd_webfs:Master
> >>> colocation fs_on_drbd inf: fs_mysql ms_drbd_mysql:Master
> >>> colocation ip_with_ip_arp inf: ip1 ip1arp
> >>> colocation mysql_on_drbd inf: MySQLDB ms_drbd_mysql:Master
> >>> colocation web_with_mysql inf: MySQLDB WebServices
> >>> colocation webfs_on_drbd inf: fs_webfs ms_drbd_webfs:Master
> >>> colocation webfs_with_fs inf: fs_webfs fs_mysql
> >>> order apache2-after-arp inf: ip1arp:start apache2:start
> >>> order arp-after-ip inf: ip1:start ip1arp:start
> >>> order fs-mysql-after-drbd inf: ms_drbd_mysql:promote fs_mysql:start
> >>> order fs-webfs-after-drbd inf: ms_drbd_webfs:promote fs_webfs:start
> >>> order ip-after-mysql inf: mysql:start ip1:start
> >>> order mysql-after-fs-mysql inf: fs_mysql:start mysql:start
> >>> property $id="cib-bootstrap-options" \
> >>> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
> >>> cluster-infrastructure="Heartbeat" \
> >>> expected-quorum-votes="1" \
> >>> stonith-enabled="false" \
> >>> no-quorum-policy="ignore"
> >>> rsc_defaults $id="rsc-options" \
> >>> resource-stickiness="100"
> >>>
> >>> Any help please?
> >>>
> >>> Thanks,
> >>>
> >>> Liang Ma
> >>> Contractuel | Consultant | SED Systems Inc.
> >>> Ground Systems Analyst
> >>> Agence spatiale canadienne | Canadian Space Agency
> >>> 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada,
> J3Y 8Y9
> >>> Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083
> >>> Courriel/E-mail : [liang.ma <http://liang.ma>@space.gc.ca
> <http://space.gc.ca>]
> >>> Site web/Web site : [www.space.gc.ca <http://www.space.gc.ca> ]
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >>>
> >>> Project Home: http://www.clusterlabs.org
> >>> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> >>> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> >>
> >> _______________________________________________
> >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >>
> >> Project Home: http://www.clusterlabs.org
> >> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> >> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> >>
> >> _______________________________________________
> >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >>
> >> Project Home: http://www.clusterlabs.org
> >> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> >> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> <mailto:Pacemaker at oss.clusterlabs.org>
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
More information about the Pacemaker
mailing list