Hi bart <br>i think your right linit is looking the best or Michael Schwartzkopff's book if its avaiable in english <br><br>Paul <br><br><div class="gmail_quote">On 1 February 2011 19:44, Bart Coninckx <span dir="ltr"><<a href="mailto:bart.coninckx@telenet.be">bart.coninckx@telenet.be</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Novell will not support you unless you use SLES, same for Redhat and<br>
RHEL. Not sure about IBM, but they are $$$. So the wonderful people of<br>
Linbit seem a good choice.<br>
<div><div></div><div class="h5"><br>
<br>
<br>
<br>
On 02/01/11 18:20, <a href="mailto:Liang.Ma@asc-csa.gc.ca">Liang.Ma@asc-csa.gc.ca</a> wrote:<br>
> Hi There,<br>
><br>
> I think we should seek professional help. Since the clusters we are going to build contain components such as heartbeat, corosync, pacemaker, drbd and ocfs2 or gfs2. And also I know there are quite a few companies who provide the support. So anyone has any suggestion, IBM, Redhat, Novell, or linbit?<br>
><br>
> Thanks in advance.<br>
><br>
> Liang Ma<br>
> Contractuel | Consultant | SED Systems Inc.<br>
> Ground Systems Analyst<br>
> Agence spatiale canadienne | Canadian Space Agency<br>
> 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9<br>
> Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083<br>
> Courriel/E-mail : [<a href="http://liang.ma" target="_blank">liang.ma</a>@<a href="http://space.gc.ca" target="_blank">space.gc.ca</a>]<br>
> Site web/Web site : [<a href="http://www.space.gc.ca" target="_blank">www.space.gc.ca</a> ]<br>
><br>
><br>
><br>
><br>
> -----Original Message-----<br>
> From: Ma, Liang<br>
> Sent: January 31, 2011 10:48 AM<br>
> To: 'The Pacemaker cluster resource manager'<br>
> Subject: RE: [Pacemaker] pacemaker won't start mysql in the second node<br>
><br>
> Sorry for sending this again. The log format was a bit messy in the previous one.<br>
><br>
> Thanks for your hints. I went through the cluster logs more carefully. By comparing the logs from the two nodes, the real different is after the line<br>
><br>
> info: process_lrm_event: LRM operation fs_mysql_start_0<br>
><br>
> On node arsvr1, after that line we got a confirmation on Action fs_mysql_start_0 as such<br>
><br>
> info: match_graph_event: Action fs_mysql_start_0 (8) confirmed on arsvr1<br>
><br>
> and then went on to Initiating action 9: start mysql_start_0 on arsvr1 (local).<br>
><br>
> However on node arsvr2, we never see the confirmation from Action fs_mysql_start_0. So mysql_start_0 is never called. But the strange thing is, I can see the drbd partition of fs_mysql is properly mounted on arsvr2. Anyone knows what might stop arsvr2 to run that Action fs_mysql_start_0 (8) confirmed?<br>
><br>
> Thanks in advance.<br>
><br>
> Here are the logs from the two nodes.<br>
><br>
> Logs on Node 2:<br>
><br>
> Jan 28 14:24:23 arsvr2 lrmd: [919]: info: rsc:fs_mysql:229: start<br>
> Jan 28 14:24:23 arsvr2 Filesystem[1568]: [1596]: INFO: Running start for /dev/drbd/by-res/r0 on /var/lib/mysql<br>
> Jan 28 14:24:23 arsvr2 lrmd: [919]: info: RA output: (fs_mysql:start:stderr) FATAL: Module scsi_hostadapter not found.<br>
> Jan 28 14:24:23 arsvr2 Filesystem[1568]: [1606]: INFO: Starting filesystem check on /dev/drbd/by-res/r0<br>
> Jan 28 14:24:23 arsvr2 lrmd: [919]: info: RA output: (fs_mysql:start:stdout) fsck from util-linux-ng 2.17.2<br>
> Jan 28 14:24:23 arsvr2 lrmd: [919]: info: RA output:(fs_mysql:start:stdout) /dev/drbd0: clean, 178/3276800 files, 257999/13106791 blocks<br>
> Jan 28 14:24:23 arsvr2 crmd: [922]: info: process_lrm_event: LRM operation fs_mysql_start_0 (call=229, rc=0, cib-update=251,confirmed=true) ok<br>
> Jan 28 14:24:46 arsvr2 cib: [918]: info: cib_stats: Processed 149 operations (0.00us average, 0% utilization) in the last 10min<br>
><br>
> Logs on Node 1:<br>
><br>
> Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: rsc:fs_mysql:867: start<br>
> Jan 28 14:28:58 arsvr1 crmd: [1068]: info: te_rsc_command: Initiating action 31: monitor drbd_mysql:1_monitor_15000 on arsvr2<br>
> Jan 28 14:28:58 arsvr1 Filesystem[516]: [544]: INFO: Running start for /dev/drbd/by-res/r0 on /var/lib/mysql<br>
> Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: RA output:(fs_mysql:start:stderr) FATAL: Module scsi_hostadapter not found.<br>
> Jan 28 14:28:58 arsvr1 Filesystem[516]: [554]: INFO: Starting filesystem check on /dev/drbd/by-res/r0<br>
> Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: RA output:(fs_mysql:start:stdout) fsck from util-linux-ng 2.17.2<br>
> Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: RA output:(fs_mysql:start:stdout) /dev/drbd0: clean, 178/3276800 files,257999/13106791 blocks<br>
> Jan 28 14:28:58 arsvr1 crmd: [1068]: info: process_lrm_event: LRM operation fs_mysql_start_0 (call=867, rc=0, cib-update=1650,confirmed=true) ok<br>
> Jan 28 14:28:58 arsvr1 crmd: [1068]: info: match_graph_event: Action fs_mysql_start_0 (8) confirmed on arsvr1 (rc=0)<br>
> Jan 28 14:28:58 arsvr1 crmd: [1068]: info: te_rsc_command: Initiating action 9: start mysql_start_0 on arsvr1 (local)<br>
> Jan 28 14:28:58 arsvr1 crmd: [1068]: info: do_lrm_rsc_op: Performing key=9:551:0:9c402121-906c-42de-a18a-68deb24208cb op=mysql_start_0 )<br>
> Jan 28 14:28:58 arsvr1 lrmd: [1065]: info: rsc:mysql:868: start<br>
> Jan 28 14:28:58 arsvr1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql<br>
> Jan 28 14:28:59 arsvr1 crmd: [1068]: info: match_graph_event: Action drbd_mysql:1_monitor_15000 (31) confirmed on arsvr2 (rc=0)<br>
> Jan 28 14:29:02 arsvr1 mysql[576]: [728]: INFO: MySQL started<br>
> Jan 28 14:29:02 arsvr1 crmd: [1068]: info: process_lrm_event: LRM operation mysql_start_0 (call=868, rc=0, cib-update=1651,confirmed=true) ok<br>
> Jan 28 14:29:02 arsvr1 crmd: [1068]: info: match_graph_event: Action mysql_start_0 (9) confirmed on arsvr1 (rc=0)<br>
><br>
><br>
> Liang Ma<br>
> Contractuel | Consultant | SED Systems Inc.<br>
> Ground Systems Analyst<br>
> Agence spatiale canadienne | Canadian Space Agency<br>
> 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9<br>
> Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083<br>
> Courriel/E-mail : [<a href="http://liang.ma" target="_blank">liang.ma</a>@<a href="http://space.gc.ca" target="_blank">space.gc.ca</a>]<br>
> Site web/Web site : [<a href="http://www.space.gc.ca" target="_blank">www.space.gc.ca</a> ]<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> -----Original Message-----<br>
> From: Dejan Muhamedagic [mailto:<a href="mailto:dejanmm@fastmail.fm">dejanmm@fastmail.fm</a>]<br>
> Sent: January 28, 2011 11:09 AM<br>
> To: The Pacemaker cluster resource manager<br>
> Subject: Re: [Pacemaker] pacemaker won't start mysql in the second node<br>
><br>
> On Fri, Jan 28, 2011 at 08:50:45AM -0500, <a href="mailto:Liang.Ma@asc-csa.gc.ca">Liang.Ma@asc-csa.gc.ca</a> wrote:<br>
>> Hi Dejan, thanks for your reply.<br>
>><br>
>> That's one of the problem. I don't see any logs in log file /var/log/mysql/error.log.<br>
><br>
> I meant the cluster logs.<br>
><br>
>> I checked the permission of directories /var/run/mysqld and /var/log/mysql. In both nodes they are the same as<br>
>><br>
>> drwxr-xr-x 2 mysql root 40 2011-01-27 13:50 /var/run/mysqld/<br>
>> drwxr-s--- 2 mysql adm 4096 2011-01-27 11:34 /var/log/mysql<br>
>><br>
>> By the way, under which user pacemaker runs, root or someone else?<br>
><br>
> pacemaker is a collection of programs. At any rate, the RA run<br>
> as root, but may su to another user (mysql) depending on the<br>
> resource configuration.<br>
><br>
> Thanks,<br>
><br>
> Dejan<br>
><br>
>> Liang Ma<br>
>> Contractuel | Consultant | SED Systems Inc.<br>
>> Ground Systems Analyst<br>
>> Agence spatiale canadienne | Canadian Space Agency<br>
>> 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9<br>
>> Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083<br>
>> Courriel/E-mail : [<a href="http://liang.ma" target="_blank">liang.ma</a>@<a href="http://space.gc.ca" target="_blank">space.gc.ca</a>]<br>
>> Site web/Web site : [<a href="http://www.space.gc.ca" target="_blank">www.space.gc.ca</a> ]<br>
>><br>
>><br>
>><br>
>><br>
>> -----Original Message-----<br>
>> From: Dejan Muhamedagic [mailto:<a href="mailto:dejanmm@fastmail.fm">dejanmm@fastmail.fm</a>]<br>
>> Sent: January 28, 2011 8:26 AM<br>
>> To: The Pacemaker cluster resource manager<br>
>> Subject: Re: [Pacemaker] pacemaker won't start mysql in the second node<br>
>><br>
>> Hi,<br>
>><br>
>> On Thu, Jan 27, 2011 at 11:51:31AM -0500, <a href="mailto:Liang.Ma@asc-csa.gc.ca">Liang.Ma@asc-csa.gc.ca</a> wrote:<br>
>>><br>
>>><br>
>>> Hi There,<br>
>>><br>
>>> I have set up a pair of ha LAMP servers using heartbeat, pacemaker and<br>
>>> drbd on Ubuntu 10.04 LTS. Everything works fine until I upgraded<br>
>>> mysql-server from 5.1.41-3ubuntu12.6 to 5.1.41-3ubuntu12.9. Now node 1<br>
>>> (arsvr1) works still fine, but mysql on node 2 (arsvr2) won't start<br>
>>> when I switch arsvr1 standby. The error message shown from "crm<br>
>>> status" is<br>
>>><br>
>>> Failed actions:<br>
>>> mysql_start_0 (node=arsvr2, call=32, rc=4, status=complete):<br>
>>> insufficient privileges<br>
>>><br>
>>> No errors logged in /var/log/mysql/error.log at all.<br>
>><br>
>> I think that you should check directory permissions. The log<br>
>> file should give you a hint.<br>
>><br>
>> Thanks,<br>
>><br>
>> Dejan<br>
>><br>
>><br>
>>> drbd mysql partition mounted properly. If I go to<br>
>>> /usr/lib/ocf/resource.d/heartbeat and set the OCF_RESKEY parameters, I<br>
>>> have no problem to start mysql server by "./mysql start". But the<br>
>>> resource mysql won't show up in crm status.<br>
>>><br>
>>> So looks somehow pacemaker fail to start resource mysql even before<br>
>>> running the resource script.<br>
>>><br>
>>> Here is the configuration<br>
>>><br>
>>> node $id="bc6bf61d-6b5f-4307-85f3-bf7bb11531bb" arsvr2 \<br>
>>> attributes standby="off"<br>
>>> node $id="bf0e7394-9684-42b9-893b-5a9a6ecddd7e" arsvr1 \<br>
>>> attributes standby="off"<br>
>>> primitive apache2 lsb:apache2 \<br>
>>> op start interval="0" timeout="60" \<br>
>>> op stop interval="0" timeout="120" start-delay="15" \<br>
>>> meta target-role="Started"<br>
>>> primitive drbd_mysql ocf:linbit:drbd \<br>
>>> params drbd_resource="r0" \<br>
>>> op monitor interval="15s"<br>
>>> primitive drbd_webfs ocf:linbit:drbd \<br>
>>> params drbd_resource="r1" \<br>
>>> op monitor interval="15s" \<br>
>>> op start interval="0" timeout="240" \<br>
>>> op stop interval="0" timeout="100"<br>
>>> primitive fs_mysql ocf:heartbeat:Filesystem \<br>
>>> params device="/dev/drbd/by-res/r0" directory="/var/lib/mysql" fstype="ext4" \<br>
>>> op start interval="0" timeout="60" \<br>
>>> op stop interval="0" timeout="120" \<br>
>>> meta target-role="Started"<br>
>>> primitive fs_webfs ocf:heartbeat:Filesystem \<br>
>>> params device="/dev/drbd/by-res/r1" directory="/srv" fstype="ext4" \<br>
>>> op start interval="0" timeout="60" \<br>
>>> op stop interval="0" timeout="120" \<br>
>>> meta target-role="Started"<br>
>>> primitive ip1 ocf:heartbeat:IPaddr2 \<br>
>>> params ip="10.10.10.193" nic="eth0" \<br>
>>> op monitor interval="5s"<br>
>>> primitive ip1arp ocf:heartbeat:SendArp \<br>
>>> params ip="10.10.10.193" nic="eth0"<br>
>>> primitive mysql ocf:heartbeat:mysql \<br>
>>> params binary="/usr/bin/mysqld_safe" config="/etc/mysql/my.cnf"<br>
>>> user="mysql" group="mysql" log="/var/log/mysql.log"<br>
>>> pid="/var/run/mysqld/mysqld.pid" datadir="/var/lib/mysql"<br>
>>> socket="/var/run/mysqld/mysqld.sock" \<br>
>>> op monitor interval="30s" timeout="30s" \<br>
>>> op start interval="0" timeout="120" \<br>
>>> op stop interval="0" timeout="120" \<br>
>>> meta target-role="Started"<br>
>>> group MySQLDB fs_mysql mysql \<br>
>>> meta target-role="Started"<br>
>>> group WebServices ip1 ip1arp fs_webfs apache2 \<br>
>>> meta target-role="Started"<br>
>>> ms ms_drbd_mysql drbd_mysql \<br>
>>> meta master-max="1" master-node-max="1" clone-max="2"<br>
>>> clone-node-max="1" notify="true"<br>
>>> ms ms_drbd_webfs drbd_webfs \<br>
>>> meta master-max="1" master-node-max="1" clone-max="2"<br>
>>> clone-node-max="1" notify="true" target-role="Started"<br>
>>> colocation apache2_with_ip inf: apache2 ip1<br>
>>> colocation apache2_with_mysql inf: apache2 ms_drbd_mysql:Master<br>
>>> colocation apache2_with_webfs inf: apache2 ms_drbd_webfs:Master<br>
>>> colocation fs_on_drbd inf: fs_mysql ms_drbd_mysql:Master<br>
>>> colocation ip_with_ip_arp inf: ip1 ip1arp<br>
>>> colocation mysql_on_drbd inf: MySQLDB ms_drbd_mysql:Master<br>
>>> colocation web_with_mysql inf: MySQLDB WebServices<br>
>>> colocation webfs_on_drbd inf: fs_webfs ms_drbd_webfs:Master<br>
>>> colocation webfs_with_fs inf: fs_webfs fs_mysql<br>
>>> order apache2-after-arp inf: ip1arp:start apache2:start<br>
>>> order arp-after-ip inf: ip1:start ip1arp:start<br>
>>> order fs-mysql-after-drbd inf: ms_drbd_mysql:promote fs_mysql:start<br>
>>> order fs-webfs-after-drbd inf: ms_drbd_webfs:promote fs_webfs:start<br>
>>> order ip-after-mysql inf: mysql:start ip1:start<br>
>>> order mysql-after-fs-mysql inf: fs_mysql:start mysql:start<br>
>>> property $id="cib-bootstrap-options" \<br>
>>> dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \<br>
>>> cluster-infrastructure="Heartbeat" \<br>
>>> expected-quorum-votes="1" \<br>
>>> stonith-enabled="false" \<br>
>>> no-quorum-policy="ignore"<br>
>>> rsc_defaults $id="rsc-options" \<br>
>>> resource-stickiness="100"<br>
>>><br>
>>> Any help please?<br>
>>><br>
>>> Thanks,<br>
>>><br>
>>> Liang Ma<br>
>>> Contractuel | Consultant | SED Systems Inc.<br>
>>> Ground Systems Analyst<br>
>>> Agence spatiale canadienne | Canadian Space Agency<br>
>>> 6767, Route de l'Aéroport, Longueuil (St-Hubert), QC, Canada, J3Y 8Y9<br>
>>> Tél/Tel : (450) 926-5099 | Téléc/Fax: (450) 926-5083<br>
>>> Courriel/E-mail : [<a href="http://liang.ma" target="_blank">liang.ma</a>@<a href="http://space.gc.ca" target="_blank">space.gc.ca</a>]<br>
>>> Site web/Web site : [<a href="http://www.space.gc.ca" target="_blank">www.space.gc.ca</a> ]<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
>>> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
>>><br>
>>> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
>>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>>> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
>><br>
>> _______________________________________________<br>
>> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
>> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
>><br>
>> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
>><br>
>> _______________________________________________<br>
>> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
>> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
>><br>
>> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
</div></div></blockquote></div><br>