[Pacemaker] Business Standard India - Errors related to pacemaker in server message file

Wed Jul 27 05:14:25 CET 2011

Dear Mr.  Mr.Andrew Beekhof,

Why errors like below which are related to pacemaker are appearing at 15
minutes interval in our message file of the server.

pengine: [3143]: ERROR: create_notification_boundaries: Creating
> > boundaries for mysql-ms-drbd

Regards,
Gururaj Patil
Systems Department
Business Standard Ltd.
Worli
Mumbai - 400013
India
Ph.+91-22-24971924

From:	pacemaker-request at oss.clusterlabs.org
To:	pacemaker at oss.clusterlabs.org
Date:	07/27/2011 08:22 AM
Subject:	Pacemaker Digest, Vol 44, Issue 54

Send Pacemaker mailing list submissions to
		 pacemaker at oss.clusterlabs.org

To subscribe or unsubscribe via the World Wide Web, visit
		 http://oss.clusterlabs.org/mailman/listinfo/pacemaker
or, via email, send a message with subject or body 'help' to
		 pacemaker-request at oss.clusterlabs.org

You can reach the person managing the list at
		 pacemaker-owner at oss.clusterlabs.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Pacemaker digest..."

Today's Topics:

   1. Re: Reload action and stop/start sequence questions
      (Andrew Beekhof)
   2. Re: Cluster type is: corosync (Andrew Beekhof)
   3. Re: Business Standard India - Errors noticed in		 pacemaker
      (Andrew Beekhof)

----------------------------------------------------------------------

Message: 1
Date: Wed, 27 Jul 2011 12:25:02 +1000
From: Andrew Beekhof <andrew at beekhof.net>
To: The Pacemaker cluster resource manager
		 <pacemaker at oss.clusterlabs.org>
Subject: Re: [Pacemaker] Reload action and stop/start sequence
		 questions
Message-ID:

<CAEDLWG0URPu_VOVzUDZ4RZoEvfeWVdEXQC1y9zQKxd0pGmErXg at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jul 11, 2011 at 5:45 PM, Vladislav Bogdanov
<bubble at hoster-ok.com> wrote:
> Hi all,
>
> Would somebody (Andrew?) please bring some light on how exactly
> redefinition of resource is supposed to be handled?
>
> Below is my (rather perfectionistic) vision on this, please correct me
> if/where I'm wrong:
> * If RA supports 'reload' action then it is called on resource
> definition change (instead of stop/start).

Only if the attribute changed was NOT marked as "unique" in the metadata.

> * If 'reload' action fails then usual start/stop sequence is executed.
> This would give a chance to RA to refuse to reload if some key
> properties change, while allowing it to tune some secondary resource
> parameters. Of course, RA should leave resource in a usable state, so
> failure of reload action should indicate RA's denial to do a reload. How
> to differentiate that from real reload failures?

Either way the resource needs to be restarted.  So there is no need
for differentiation.

> Is there some special
> exit code for that?
> * Dependent resources should not be stopped/started for 'reload' action.
> Of course they are restarted if reload fails and stop/start is executed
> then. (I see that they are restarted now for reload of a resource they
> depend on, is it a bug?)

More like a limitation.  Which is a round-a-bout way of saying "really
hard to fix bug".
You're welcome to create a BZ for it though, maybe one day I'll figure
out how to resolve it.

> * (wish) Resources should be migrated out of node (if they support live
> migration) for stop/start sequence of resource they depend on.

Migration can only occur if a resource at the bottom (excluding any
clones) of the resource stack.
In order to migrate any colocation dependancies need to be running at
_both_ the old and the new locations.

This can only be true for resources that depend on clones.

> * (wish) Redefinition of clones should be handled in a way which allows
> dependent live-migratable resources to survive (if reload action for
> clone instance either is not supported or fails).

This doesn't make sense.
If the definition of one clone changes, then they all change and there
is nowhere for dependant resources to migrate to.

> That is: dependent
> resources which support live migration are first tried to migrate out of
> one node, and are stopped if migration fails. Then clone instance is
> restarted on that node. Then the same procedure applies to next cluster
> node so resources may return back to a first node.
>
> If above (at least first three points) is right, then is it possible to
> get a set of previous instance parameters the same way new configuration
> is passed (env vars), or RA should save that information itself in
advance?
>
> Best,
> Vladislav
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>

------------------------------

Message: 2
Date: Wed, 27 Jul 2011 12:41:35 +1000
From: Andrew Beekhof <andrew at beekhof.net>
To: The Pacemaker cluster resource manager
		 <pacemaker at oss.clusterlabs.org>
Subject: Re: [Pacemaker] Cluster type is: corosync
Message-ID:

<CAEDLWG3rcFDM1ew8eTne5U2R25QL3rkoS5ZFY_oM6OHxZtL0QA at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

On Tue, Jul 26, 2011 at 5:12 PM, Proskurin Kirill
<k.proskurin at corp.mail.ru> wrote:
> On 07/26/2011 11:00 AM, Andrew Beekhof wrote:
>>
>> On Mon, Jul 25, 2011 at 7:18 PM, Proskurin Kirill
>> <k.proskurin at corp.example.com> ?wrote:
>>>
>>> 25.07.2011 10:10, Andrew Beekhof ?????:
>>>>
>>>> Which packages are you using?
>>>
>>> It is your official source from repository I build.
>>
>> Ok. And did you add the pacemaker configuration options to corosync's
>> config file?
>
>
> I attach our corosync.conf. It is same on all nodes except IP addr.

You missed a step from:

http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/s-configure-corosync.html

> Pacemaker is black now - no configuration at all.
>
> Online nodes:
> [root at mysender1 ~]# crm configure show
> node mysender1.example.com
> node mysender2.example.com
> node mysender3.example.com
> node mysender4.example.com
> node mysender5.example.com
> node mysender6.example.com
> node mysender7.example.com
> property $id="cib-bootstrap-options" \
> ? ? ? ?dc-version="1.1.5-3-01e86afaaa6d4a8c4836f68df80ababd6ca3902f" \
> ? ? ? ?cluster-infrastructure="openais" \
> ? ? ? ?expected-quorum-votes="6"
>
>
> Offline nodes(Cluster type is: corosync)
> [root at mysender2 ~]# crm configure show
> [root at mysender2 ~]#
>
>
>>
>>> pacemaker-1.1.5
>>> corosync-1.4.0
>>> cluster-glue-1.0.6
>>> openais-1.1.2
>>>
>>> All nodes have same rpms.
>>>
>>>> On Fri, Jul 22, 2011 at 7:47 PM, Proskurin Kirill
>>>> <k.proskurin at corp.example.com> ? ?wrote:
>>>>>
>>>>> Hello again!
>>>>>
>>>>> Hope I`m not flooding too much here but I have another problem.
>>>>>
>>>>> I install same rpm of corosync, openais, pacemaker, cluster_glue on
all
>>>>> nodes. I check it twice.
>>>>>
>>>>> And then I start some of they - they can`t connect to cluster and
stays
>>>>> offline. In logs I see what they see other nodes and connectivity is
>>>>> ok.
>>>>> But
>>>>> I found the difference:
>>>>>
>>>>> Online nodes in cluster have:
>>>>> [root at mysender39 ~]# grep 'Cluster type is' /var/log/corosync.log
>>>>> Jul 22 20:38:58 mysender39.example.com stonith-ng: [3499]: info:
>>>>> get_cluster_type: Cluster type is: 'openais'.
>>>>> Jul 22 20:38:58 mysender39.example.com attrd: [3502]: info:
>>>>> get_cluster_type:
>>>>> Cluster type is: 'openais'.
>>>>> Jul 22 20:38:58 mysender39.example.com cib: [3500]: info:
>>>>> get_cluster_type:
>>>>> Cluster type is: 'openais'.
>>>>> Jul 22 20:38:59 mysender39.example.com crmd: [3504]: info:
>>>>> get_cluster_type:
>>>>> Cluster type is: 'openais'.
>>>>>
>>>>> Offline have:
>>>>> [root at mysender2 ~]# grep 'Cluster type is' /var/log/corosync.log
>>>>> Jul 22 13:39:17 mysender2.example.com stonith-ng: [9028]: info:
>>>>> get_cluster_type: Cluster type is: 'corosync'.
>>>>> Jul 22 13:39:17 mysender2.example.com attrd: [9031]: info:
>>>>> get_cluster_type:
>>>>> Cluster type is: 'corosync'.
>>>>> Jul 22 13:39:17 mysender2.example.com cib: [9029]: info:
>>>>> get_cluster_type:
>>>>> Cluster type is: 'corosync'.
>>>>> Jul 22 13:39:18 mysender2.example.com crmd: [9033]: info:
>>>>> get_cluster_type:
>>>>> Cluster type is: 'corosync'.
>>>>>
>>>>> What`s wrong and how can I fix it?
>
> --
> Best regards,
> Proskurin Kirill
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>

------------------------------

Message: 3
Date: Wed, 27 Jul 2011 12:48:28 +1000
From: Andrew Beekhof <andrew at beekhof.net>
To: The Pacemaker cluster resource manager
		 <pacemaker at oss.clusterlabs.org>
Cc: Prafulla H Patil <prafulla.patil at bsmail.in>,		 Bandana Roy
		 <bandana.roy at bsmail.in>
Subject: Re: [Pacemaker] Business Standard India - Errors noticed in
		 pacemaker
Message-ID:
		 <CAEDLWG2t+hnb9N5DjY3gAm5Xw84JxQTQtrP
+un1QK=BFv4pL_g at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

On Tue, Jul 26, 2011 at 5:35 PM, Gururaj B Patil
<gururaj.patil at bsmail.in>wrote:

> Dear Mr.Andrew Beekhof
>
> Yes I did try googeling but could not get proper information. In few
forums
> I noticed message as below.
>
> "That's development logging, which was accidentally bumped to a higher
> log level."
>

What was unclear about this?

>
>
> We are searching again but meanwhile can pacemaker team help on this.
>
> Regards,
> Gururaj Patil
>
> [image: Inactive hide details for pacemaker-request---07/26/2011 12:47:15
> PM---Send Pacemaker mailing list submissions to
pacemaker at os]pacemaker-request---07/26/2011
> 12:47:15 PM---Send Pacemaker mailing list submissions to
> pacemaker at oss.clusterlabs.org
>
> From: pacemaker-request at oss.clusterlabs.org
> To: pacemaker at oss.clusterlabs.org
> Date: 07/26/2011 12:47 PM
> Subject: Pacemaker Digest, Vol 44, Issue 50
> ------------------------------
>
>
>
> Send Pacemaker mailing list submissions to
> pacemaker at oss.clusterlabs.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> or, via email, send a message with subject or body 'help' to
> pacemaker-request at oss.clusterlabs.org
>
> You can reach the person managing the list at
> pacemaker-owner at oss.clusterlabs.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Pacemaker digest..."
>
>
> Today's Topics:
>
>   1. Re: Business Standard India - Errors noticed in pacemaker
>      (Andrew Beekhof)
>   2. Re: Cluster type is: corosync (Andrew Beekhof)
>   3. Please teach it about handling of the unmanaged resource in
>      environment setting placement-strategy. (Yuusuke IIDA)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 26 Jul 2011 16:02:30 +1000
> From: Andrew Beekhof <andrew at beekhof.net>
> To: The Pacemaker cluster resource manager
> <pacemaker at oss.clusterlabs.org>
> Cc: M A Faruqui <m.faruqui at bsmail.in>, Prafulla H Patil
> <prafulla.patil at bsmail.in>, Bandana Roy <bandana.roy at bsmail.in>
> Subject: Re: [Pacemaker] Business Standard India - Errors noticed in
> pacemaker
> Message-ID:
> <CAEDLWG0ExctnrJq8KduV8JdC3hZTVSovB7gDnYv4MVpPox_cBw at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Thought about googling the error?
>
> On Mon, Jul 25, 2011 at 9:17 PM, Gururaj B Patil <gururaj.patil at bsmail.in
> >wrote:
>
> > To support staff,
> >
> > We in Business Standard Ltd. use pacemaker as clusttering application
for
> > one of our website. Two servers are are in clusttering mode.
> >
> > One of the server is web server and another one is mysql db server.
> > Pacemaker handles Mysql clustering at block level.
> >
> > We have noticed same type of notice and warning in the server's message
> > file. Errors are as below.
> >
> >
> >
>
-----------------------------------------------------------------------------------------------------------------------------------------------

> > Messages like below appear every 15 minutes
> >
> > sidrbd0 pengine: [3143]: notice: get_failcount: Failcount for sipdu1 on
> > sidrbd0 has expired (limit was 20s)
> >
> > sidrbd0 pengine: [3143]: ERROR: create_notification_boundaries:
Creating
> > boundaries for mysql-ms-drbd
> >
> >
> >
>
-----------------------------------------------------------------------------------------------------------------------------------------------

> >
> > I have registered for pacemaker mailing list also.
> >
> > Regards,
> > Gururaj Patil
> > Systems Department
> > Business Standard Ltd.
> > H3/4, Paragon center,
> > P.B.Marg,
> > Worli
> > Mumbai - 400013
> > India
> >
> > Ph.+91-22-24971924
> > ------------------------------
> >
> >
> >
> > *Disclaimer:* "This communication/message is for the named addressees
> > only. This transmission may contain information that is privileged,
> > confidential, proprietary or legally privileged, and /or exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> > please immediately notify the sender and destroy the material in its
> > entirety, whether in electronic or hard copy format. You are hereby
> notified
> > that any disclosure, copying, distribution, or use of the information
> > contained herein (including any reliance thereon) is STRICTLY
> PROBHIBITED.
> > You must not, directly or indirectly, use, disclose, distribute, print
or
> > copy any part of this message."
> >
> > *WARNING :*"This electronic mail and any attachments are believed to be
> > free of any virus or other defect, the recipient must ensure that it is
> > virus free and no responsibility is accepted by Business Standard
Limited
> > and /or its employees as applicable for any loss or damage arising in
any
> > way from its use."
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs:
> >
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
>
http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110726/4d99749e/attachment-0001.html

> >
>
> ------------------------------
>
> Message: 2
> Date: Tue, 26 Jul 2011 17:00:56 +1000
> From: Andrew Beekhof <andrew at beekhof.net>
> To: Pacemaker at oss.clusterlabs.org
> Subject: Re: [Pacemaker] Cluster type is: corosync
> Message-ID:
> <CAEDLWG1s=AHrdXdWdO0r04410j0+Ygj7VfAz_Yf_0fMDpiNorA at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Mon, Jul 25, 2011 at 7:18 PM, Proskurin Kirill
> <k.proskurin at corp.mail.ru> wrote:
> > 25.07.2011 10:10, Andrew Beekhof ?????:
> >>
> >> Which packages are you using?
> >
> > It is your official source from repository I build.
>
> Ok. And did you add the pacemaker configuration options to corosync's
> config file?
>
> > pacemaker-1.1.5
> > corosync-1.4.0
> > cluster-glue-1.0.6
> > openais-1.1.2
> >
> > All nodes have same rpms.
> >
> >> On Fri, Jul 22, 2011 at 7:47 PM, Proskurin Kirill
> >> <k.proskurin at corp.mail.ru> ?wrote:
> >>>
> >>> Hello again!
> >>>
> >>> Hope I`m not flooding too much here but I have another problem.
> >>>
> >>> I install same rpm of corosync, openais, pacemaker, cluster_glue on
all
> >>> nodes. I check it twice.
> >>>
> >>> And then I start some of they - they can`t connect to cluster and
stays
> >>> offline. In logs I see what they see other nodes and connectivity is
> ok.
> >>> But
> >>> I found the difference:
> >>>
> >>> Online nodes in cluster have:
> >>> [root at mysender39 ~]# grep 'Cluster type is' /var/log/corosync.log
> >>> Jul 22 20:38:58 mysender39.mail.ru stonith-ng: [3499]: info:
> >>> get_cluster_type: Cluster type is: 'openais'.
> >>> Jul 22 20:38:58 mysender39.mail.ru attrd: [3502]: info:
> get_cluster_type:
> >>> Cluster type is: 'openais'.
> >>> Jul 22 20:38:58 mysender39.mail.ru cib: [3500]: info:
> get_cluster_type:
> >>> Cluster type is: 'openais'.
> >>> Jul 22 20:38:59 mysender39.mail.ru crmd: [3504]: info:
> get_cluster_type:
> >>> Cluster type is: 'openais'.
> >>>
> >>> Offline have:
> >>> [root at mysender2 ~]# grep 'Cluster type is' /var/log/corosync.log
> >>> Jul 22 13:39:17 mysender2.mail.ru stonith-ng: [9028]: info:
> >>> get_cluster_type: Cluster type is: 'corosync'.
> >>> Jul 22 13:39:17 mysender2.mail.ru attrd: [9031]: info:
> get_cluster_type:
> >>> Cluster type is: 'corosync'.
> >>> Jul 22 13:39:17 mysender2.mail.ru cib: [9029]: info:
get_cluster_type:
> >>> Cluster type is: 'corosync'.
> >>> Jul 22 13:39:18 mysender2.mail.ru crmd: [9033]: info:
> get_cluster_type:
> >>> Cluster type is: 'corosync'.
> >>>
> >>> What`s wrong and how can I fix it?
> >
> > --
> > Best regards,
> > Proskurin Kirill
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs:
> >
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
> >
>
>
>
> ------------------------------
>
> Message: 3
> Date: Tue, 26 Jul 2011 16:12:25 +0900
> From: Yuusuke IIDA <iidayuus at intellilink.co.jp>
> To: "pacemaker at oss" <pacemaker at oss.clusterlabs.org>
> Cc: tanakakza at intellilink.co.jp
> Subject: [Pacemaker] Please teach it about handling of the unmanaged
> resource in environment setting placement-strategy.
> Message-ID: <4E2E68D9.6010308 at intellilink.co.jp>
> Content-Type: text/plain; charset="iso-2022-jp"
>
> Hi, Yan
> Hi, Andrew
>
> I used the function of placement-strategy and found movement to be
worried
> about.
>
> There is node "act3" which the resource that became the unmanaged state
> starts.
>
> The resource that started then in node "act1" broke down and moved.
>
> I hoped that this inoperative resource moved to node "sby1", but it was
not
> carried out.
>
> Is the movement that a resource with other capacity moves in the node
that
> the
> resource of the unmanaged state meets capacity right as specifications?
>
> I want you to revise it to decide placement in consideration of the
> capacity of
> the unmanaged resource.
>
> I attach crm_report when a problem happened.
>
> Best Regards,
> Yuusuke
> --
> ----------------------------------------
> METRO SYSTEMS CO., LTD
>
> Yuusuke Iida
> Mail: iidayuus at intellilink.co.jp
> ----------------------------------------
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: pcmk-Tue-26-Jul-2011.tar.bz2
> Type: application/octet-stream
> Size: 164949 bytes
> Desc: not available
> URL: <
>
http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110726/66852f5b/attachment.obj

> >
>
> ------------------------------
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
>
> End of Pacemaker Digest, Vol 44, Issue 50
> *****************************************
>
>
> ------------------------------
>
>
>
> *Disclaimer:* "This communication/message is for the named addressees
> only. This transmission may contain information that is privileged,
> confidential, proprietary or legally privileged, and /or exempt from
> disclosure under applicable law. If you are not the intended recipient,
> please immediately notify the sender and destroy the material in its
> entirety, whether in electronic or hard copy format. You are hereby
notified
> that any disclosure, copying, distribution, or use of the information
> contained herein (including any reliance thereon) is STRICTLY
PROBHIBITED.
> You must not, directly or indirectly, use, disclose, distribute, print or
> copy any part of this message."
>
> *WARNING :*"This electronic mail and any attachments are believed to be
> free of any virus or other defect, the recipient must ensure that it is
> virus free and no responsibility is accepted by Business Standard Limited
> and /or its employees as applicable for any loss or damage arising in any
> way from its use."
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110727/77cdae26/attachment.html
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <
http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110727/77cdae26/attachment.gif
>

------------------------------

_______________________________________________
Pacemaker mailing list
Pacemaker at oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

End of Pacemaker Digest, Vol 44, Issue 54
*****************************************

Disclaimer: "This communication/message is for the named addressees only. This transmission may contain information that is privileged, confidential, proprietary or legally privileged, and /or exempt from disclosure under applicable law. If you are not the intended recipient, please immediately notify the sender and destroy the material in its entirety, whether in electronic or hard copy format. You are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROBHIBITED. You must not, directly or indirectly, use, disclose, distribute, print or copy any part of this message."

 WARNING :"This electronic mail and any attachments are believed to be free of any virus or other defect, the recipient must ensure that it is virus free and no responsibility is accepted by Business Standard Limited and /or its employees as applicable for any loss or damage arising in any way from its use."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110727/2d2ea6d6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20110727/2d2ea6d6/attachment-0001.gif>