<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.16.3">
</HEAD>
<BODY>
Hi Fabio,<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<TT>the gfs2 example was not</TT> ....<BR>
</BLOCKQUOTE>
<BR>
You are forgiven ... and you are light years ahead of me. I have folks after my docs, and have not had time to convert notes to docs. Top of my to-do list after this task.<BR>
<BR>
I have been fumbling getting a combination of "chkconfig blah on" and Pacemaker resources that work. I'll go back to the "chkconfig clvmd on" combination for another try ... nothing like learning that this works to get me to try it again!<BR>
<BR>
Yep, running the latest RHEL6.5 updates. Learn (years ago) to compulsively check for updates when working with HA software.<BR>
<BR>
I'll let you know how this goes.<BR>
<BR>
Thanks,<BR>
Bob Haxo<BR>
<BR>
<BR>
On Tue, 2014-01-07 at 09:21 +0100, Fabio M. Di Nitto wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">On 1/6/2014 6:24 PM, Bob Haxo wrote:</FONT>
<FONT COLOR="#000000">> Hi Fabio,</FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">>>> There is an example on how to configure gfs2 also in the rhel6.5</FONT>
<FONT COLOR="#000000">>>> pacemaker documentation, using pcs.</FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> Super! Please share the link to this documentation. I only discovered</FONT>
<FONT COLOR="#000000">> the gfs2+pcs example with the rhel7 beta docs.</FONT>
<FONT COLOR="#000000">You are right, the gfs2 example was not published in Rev 1 of the</FONT>
<FONT COLOR="#000000">pacemaker documentation for RHEL6.5. It´s entirely possible I missed it</FONT>
<FONT COLOR="#000000">during doc review, sorry about that!</FONT>
<FONT COLOR="#000000"><A HREF="https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Configuring_the_Red_Hat_High_Availability_Add-On_with_Pacemaker/index.html">https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Configuring_the_Red_Hat_High_Availability_Add-On_with_Pacemaker/index.html</A></FONT>
<FONT COLOR="#000000">Short version is:</FONT>
<FONT COLOR="#000000">chkconfig cman on</FONT>
<FONT COLOR="#000000">chkconfig clvmd on</FONT>
<FONT COLOR="#000000">chkconfig pacemaker on</FONT>
<FONT COLOR="#000000">Use the above doc to setup / start the cluster (stop after stonith config)</FONT>
<FONT COLOR="#000000">Setup your clvmd storage (note that neither dlm or clvmd are managed by</FONT>
<FONT COLOR="#000000">pacemaker in RHEL6.5 vs RHEL7 where it´s all managed by pacemaker).</FONT>
<FONT COLOR="#000000">Start adding your resources/services here etc...</FONT>
<FONT COLOR="#000000">Also, make absolutely sure you have all the latest updates from 6.5</FONT>
<FONT COLOR="#000000">Erratas installed.</FONT>
<FONT COLOR="#000000">Fabio</FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> Bob Haxo</FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> On Sat, 2014-01-04 at 16:56 +0100, Fabio M. Di Nitto wrote:</FONT>
<FONT COLOR="#000000">>> On 01/01/2014 01:57 AM, Bob Haxo wrote:</FONT>
<FONT COLOR="#000000">>> > Greetings ... Happy New Year!</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > I am testing a configuration that is created from example in "Chapter 6.</FONT>
<FONT COLOR="#000000">>> > Configuring a GFS2 File System in a Cluster" of the "Red Hat Enterprise</FONT>
<FONT COLOR="#000000">>> > Linux 7.0 Beta Global File System 2" document. Only addition is</FONT>
<FONT COLOR="#000000">>> > stonith:fence_ipmilan. After encountering this issue when I configured</FONT>
<FONT COLOR="#000000">>> > with "crm", I re-configured using "pcs". I've included the configuration</FONT>
<FONT COLOR="#000000">>> > below.</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> Hold on a second here.. why are you using RHEL7 documentation to</FONT>
<FONT COLOR="#000000">>> configure RHEL6.5? Please don't mix :) there are some differences and we</FONT>
<FONT COLOR="#000000">>> definitely never tested mixing those up.</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> There is an example on how to configure gfs2 also in the rhel6.5</FONT>
<FONT COLOR="#000000">>> pacemaker documentation, using pcs.</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> I personally never saw this behaviour, so it's entirely possible that</FONT>
<FONT COLOR="#000000">>> mixing things up will result in unpredictable status.</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> Fabio</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > I'm thinking that, in a 2-node cluster, if I run "stonith_admin -F</FONT>
<FONT COLOR="#000000">>> > <peer-node>", then <peer-node> should reboot and cleanly rejoin the</FONT>
<FONT COLOR="#000000">>> > cluster. This is not happening. </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > What ultimately happens is that after the initially fenced node reboots,</FONT>
<FONT COLOR="#000000">>> > the system from which the stonith_admin -F command was run is fenced and</FONT>
<FONT COLOR="#000000">>> > reboots. The fencing stops there, leaving the cluster in an appropriate</FONT>
<FONT COLOR="#000000">>> > state.</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > The issue seems to reside with clvmd/lvm. With the reboot of the</FONT>
<FONT COLOR="#000000">>> > initially fenced node, the clvmd resource fails on the surviving node,</FONT>
<FONT COLOR="#000000">>> > with a maximum of errors. I hypothesize there is an issue with locks,</FONT>
<FONT COLOR="#000000">>> > but have insufficient knowledge of clvmd/lvm locks to prove or disprove</FONT>
<FONT COLOR="#000000">>> > this hypothesis.</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Have I missed something ...</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > 1) Is this expected behavior, and always the reboot of the fencing node</FONT>
<FONT COLOR="#000000">>> > happens?</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > 2) Or, maybe I didn't correctly duplicate the Chapter 6 example?</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > 3) Or, perhaps something is wrong or omitted from the Chapter 6 example?</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Suggestions will be much appreciated.</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Thanks,</FONT>
<FONT COLOR="#000000">>> > Bob Haxo</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > RHEL6.5</FONT>
<FONT COLOR="#000000">>> > pacemaker-cli-1.1.10-14.el6_5.1.x86_64</FONT>
<FONT COLOR="#000000">>> > crmsh-1.2.5-55.1sgi709r3.rhel6.x86_64</FONT>
<FONT COLOR="#000000">>> > pacemaker-libs-1.1.10-14.el6_5.1.x86_64</FONT>
<FONT COLOR="#000000">>> > cman-3.0.12.1-59.el6_5.1.x86_64</FONT>
<FONT COLOR="#000000">>> > pacemaker-1.1.10-14.el6_5.1.x86_64</FONT>
<FONT COLOR="#000000">>> > corosynclib-1.4.1-17.el6.x86_64</FONT>
<FONT COLOR="#000000">>> > corosync-1.4.1-17.el6.x86_64</FONT>
<FONT COLOR="#000000">>> > pacemaker-cluster-libs-1.1.10-14.el6_5.1.x86_64</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Cluster Name: mici</FONT>
<FONT COLOR="#000000">>> > Corosync Nodes:</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Pacemaker Nodes:</FONT>
<FONT COLOR="#000000">>> > mici-admin mici-admin2</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Resources:</FONT>
<FONT COLOR="#000000">>> > Clone: clusterfs-clone</FONT>
<FONT COLOR="#000000">>> > Meta Attrs: interleave=true target-role=Started</FONT>
<FONT COLOR="#000000">>> > Resource: clusterfs (class=ocf provider=heartbeat type=Filesystem)</FONT>
<FONT COLOR="#000000">>> > Attributes: device=/dev/vgha2/lv_clust2 directory=/images fstype=gfs2</FONT>
<FONT COLOR="#000000">>> > options=defaults,noatime,nodiratime</FONT>
<FONT COLOR="#000000">>> > Operations: monitor on-fail=fence interval=30s</FONT>
<FONT COLOR="#000000">>> > (clusterfs-monitor-interval-30s)</FONT>
<FONT COLOR="#000000">>> > Clone: clvmd-clone</FONT>
<FONT COLOR="#000000">>> > Meta Attrs: interleave=true ordered=true target-role=Started</FONT>
<FONT COLOR="#000000">>> > Resource: clvmd (class=lsb type=clvmd)</FONT>
<FONT COLOR="#000000">>> > Operations: monitor on-fail=fence interval=30s</FONT>
<FONT COLOR="#000000">>> > (clvmd-monitor-interval-30s)</FONT>
<FONT COLOR="#000000">>> > Clone: dlm-clone</FONT>
<FONT COLOR="#000000">>> > Meta Attrs: interleave=true ordered=true</FONT>
<FONT COLOR="#000000">>> > Resource: dlm (class=ocf provider=pacemaker type=controld)</FONT>
<FONT COLOR="#000000">>> > Operations: monitor on-fail=fence interval=30s (dlm-monitor-interval-30s)</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Stonith Devices:</FONT>
<FONT COLOR="#000000">>> > Resource: p_ipmi_fencing_1 (class=stonith type=fence_ipmilan)</FONT>
<FONT COLOR="#000000">>> > Attributes: ipaddr=128.##.##.78 login=XXXXX passwd=XXXXX lanplus=1</FONT>
<FONT COLOR="#000000">>> > action=reboot pcmk_host_check=static-list pcmk_host_list=mici-admin</FONT>
<FONT COLOR="#000000">>> > Meta Attrs: target-role=Started</FONT>
<FONT COLOR="#000000">>> > Operations: monitor start-delay=30 interval=60s timeout=30</FONT>
<FONT COLOR="#000000">>> > (p_ipmi_fencing_1-monitor-60s)</FONT>
<FONT COLOR="#000000">>> > Resource: p_ipmi_fencing_2 (class=stonith type=fence_ipmilan)</FONT>
<FONT COLOR="#000000">>> > Attributes: ipaddr=128.##.##.220 login=XXXXX passwd=XXXXX lanplus=1</FONT>
<FONT COLOR="#000000">>> > action=reboot pcmk_host_check=static-list pcmk_host_list=mici-admin2</FONT>
<FONT COLOR="#000000">>> > Meta Attrs: target-role=Started</FONT>
<FONT COLOR="#000000">>> > Operations: monitor start-delay=30 interval=60s timeout=30</FONT>
<FONT COLOR="#000000">>> > (p_ipmi_fencing_2-monitor-60s)</FONT>
<FONT COLOR="#000000">>> > Fencing Levels:</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Location Constraints:</FONT>
<FONT COLOR="#000000">>> > Resource: p_ipmi_fencing_1</FONT>
<FONT COLOR="#000000">>> > Disabled on: mici-admin (score:-INFINITY)</FONT>
<FONT COLOR="#000000">>> > (id:location-p_ipmi_fencing_1-mici-admin--INFINITY)</FONT>
<FONT COLOR="#000000">>> > Resource: p_ipmi_fencing_2</FONT>
<FONT COLOR="#000000">>> > Disabled on: mici-admin2 (score:-INFINITY)</FONT>
<FONT COLOR="#000000">>> > (id:location-p_ipmi_fencing_2-mici-admin2--INFINITY)</FONT>
<FONT COLOR="#000000">>> > Ordering Constraints:</FONT>
<FONT COLOR="#000000">>> > start dlm-clone then start clvmd-clone (Mandatory)</FONT>
<FONT COLOR="#000000">>> > (id:order-dlm-clone-clvmd-clone-mandatory)</FONT>
<FONT COLOR="#000000">>> > start clvmd-clone then start clusterfs-clone (Mandatory)</FONT>
<FONT COLOR="#000000">>> > (id:order-clvmd-clone-clusterfs-clone-mandatory)</FONT>
<FONT COLOR="#000000">>> > Colocation Constraints:</FONT>
<FONT COLOR="#000000">>> > clusterfs-clone with clvmd-clone (INFINITY)</FONT>
<FONT COLOR="#000000">>> > (id:colocation-clusterfs-clone-clvmd-clone-INFINITY)</FONT>
<FONT COLOR="#000000">>> > clvmd-clone with dlm-clone (INFINITY)</FONT>
<FONT COLOR="#000000">>> > (id:colocation-clvmd-clone-dlm-clone-INFINITY)</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Cluster Properties:</FONT>
<FONT COLOR="#000000">>> > cluster-infrastructure: cman</FONT>
<FONT COLOR="#000000">>> > dc-version: 1.1.10-14.el6_5.1-368c726</FONT>
<FONT COLOR="#000000">>> > last-lrm-refresh: 1388530552</FONT>
<FONT COLOR="#000000">>> > no-quorum-policy: ignore</FONT>
<FONT COLOR="#000000">>> > stonith-enabled: true</FONT>
<FONT COLOR="#000000">>> > Node Attributes:</FONT>
<FONT COLOR="#000000">>> > mici-admin: standby=off</FONT>
<FONT COLOR="#000000">>> > mici-admin2: standby=off</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Last updated: Tue Dec 31 17:15:55 2013</FONT>
<FONT COLOR="#000000">>> > Last change: Tue Dec 31 16:57:37 2013 via cibadmin on mici-admin</FONT>
<FONT COLOR="#000000">>> > Stack: cman</FONT>
<FONT COLOR="#000000">>> > Current DC: mici-admin2 - partition with quorum</FONT>
<FONT COLOR="#000000">>> > Version: 1.1.10-14.el6_5.1-368c726</FONT>
<FONT COLOR="#000000">>> > 2 Nodes configured</FONT>
<FONT COLOR="#000000">>> > 8 Resources configured</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Online: [ mici-admin mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Full list of resources:</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > p_ipmi_fencing_1 (stonith:fence_ipmilan): Started mici-admin2</FONT>
<FONT COLOR="#000000">>> > p_ipmi_fencing_2 (stonith:fence_ipmilan): Started mici-admin</FONT>
<FONT COLOR="#000000">>> > Clone Set: clusterfs-clone [clusterfs]</FONT>
<FONT COLOR="#000000">>> > Started: [ mici-admin mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > Clone Set: clvmd-clone [clvmd]</FONT>
<FONT COLOR="#000000">>> > Started: [ mici-admin mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > Clone Set: dlm-clone [dlm]</FONT>
<FONT COLOR="#000000">>> > Started: [ mici-admin mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Migration summary:</FONT>
<FONT COLOR="#000000">>> > * Node mici-admin:</FONT>
<FONT COLOR="#000000">>> > * Node mici-admin2:</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > =====================================================</FONT>
<FONT COLOR="#000000">>> > crm_mon after the fenced node reboots. Shows the failure of clvmd that</FONT>
<FONT COLOR="#000000">>> > then</FONT>
<FONT COLOR="#000000">>> > occurs, which in turn triggers a fencing of that nnode</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Last updated: Tue Dec 31 17:06:55 2013</FONT>
<FONT COLOR="#000000">>> > Last change: Tue Dec 31 16:57:37 2013 via cibadmin on mici-admin</FONT>
<FONT COLOR="#000000">>> > Stack: cman</FONT>
<FONT COLOR="#000000">>> > Current DC: mici-admin - partition with quorum</FONT>
<FONT COLOR="#000000">>> > Version: 1.1.10-14.el6_5.1-368c726</FONT>
<FONT COLOR="#000000">>> > 2 Nodes configured</FONT>
<FONT COLOR="#000000">>> > 8 Resources configured</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Node mici-admin: UNCLEAN (online)</FONT>
<FONT COLOR="#000000">>> > Online: [ mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Full list of resources:</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > p_ipmi_fencing_1 (stonith:fence_ipmilan): Stopped</FONT>
<FONT COLOR="#000000">>> > p_ipmi_fencing_2 (stonith:fence_ipmilan): Started mici-admin</FONT>
<FONT COLOR="#000000">>> > Clone Set: clusterfs-clone [clusterfs]</FONT>
<FONT COLOR="#000000">>> > Started: [ mici-admin ]</FONT>
<FONT COLOR="#000000">>> > Stopped: [ mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > Clone Set: clvmd-clone [clvmd]</FONT>
<FONT COLOR="#000000">>> > clvmd (lsb:clvmd): FAILED mici-admin</FONT>
<FONT COLOR="#000000">>> > Stopped: [ mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > Clone Set: dlm-clone [dlm]</FONT>
<FONT COLOR="#000000">>> > Started: [ mici-admin mici-admin2 ]</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Migration summary:</FONT>
<FONT COLOR="#000000">>> > * Node mici-admin:</FONT>
<FONT COLOR="#000000">>> > clvmd: migration-threshold=1000000 fail-count=1 last-failure='Tue Dec</FONT>
<FONT COLOR="#000000">>> > 31 17:04:29 2013'</FONT>
<FONT COLOR="#000000">>> > * Node mici-admin2:</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Failed actions:</FONT>
<FONT COLOR="#000000">>> > clvmd_monitor_30000 on mici-admin 'unknown error' (1): call=60,</FONT>
<FONT COLOR="#000000">>> > status=Timed Out, la</FONT>
<FONT COLOR="#000000">>> > st-rc-change='Tue Dec 31 17:04:29 2013', queued=0ms, exec=0ms</FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > _______________________________________________</FONT>
<FONT COLOR="#000000">>> > Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A> <<A HREF="mailto:Pacemaker@oss.clusterlabs.org">mailto:Pacemaker@oss.clusterlabs.org</A>></FONT>
<FONT COLOR="#000000">>> > <A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>> > Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">>> > Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">>> > Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">>> > </FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> _______________________________________________</FONT>
<FONT COLOR="#000000">>> Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A> <<A HREF="mailto:Pacemaker@oss.clusterlabs.org">mailto:Pacemaker@oss.clusterlabs.org</A>></FONT>
<FONT COLOR="#000000">>> <A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">>> Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">>> Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> _______________________________________________</FONT>
<FONT COLOR="#000000">> Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">> <A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">> Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">> Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">> Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">> </FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>