No subject
Tue Oct 23 06:10:09 UTC 2012
the promotion of the new master even though it does not have the latest dat=
a.
If this is correct is there a way to force the promotion of the node with t=
he newest data?
v/r
STEVE
On Mar 26, 2013, at 8:19 AM, Steven Bambling <smbambling at arin.net<mailto:sm=
bambling at arin.net>> wrote:
Excellent thanks so much for the clarification. I'll drop this new RA in a=
nd see if I can get things working.
STEVE
On Mar 26, 2013, at 7:38 AM, Rainer Brestan <rainer.brestan at gmx.net<mailto:=
rainer.brestan at gmx.net>>
wrote:
Hi Steve,
pgsql RA does the same, it compares the last_xlog_replay_location of all no=
des for master promotion.
Doing a promote as a restart instead of promote command to conserve timelin=
e id is also on configurable option (restart_on_promote) of the current RA.
And the RA is definitely capable of having more than two instances. It goes=
through the parameter node_list and doing its actions for every member in =
the node list.
Originally it might be planned only to have only one slave, but the current=
implementation does not have this limitation. It has code for sync replica=
tion of more than two nodes, when some of them fall back into async to not =
promote them.
Of course, i will share the extension with the community, when they are rea=
dy for use. And the feature of having more than two instances is not remove=
d. I am not running more than two instances on one site, current usage is t=
o have two instances on one site and having two sites and manage master by =
booth. But it also under discussion to have more than two instances on one =
site, just to have no availability interruption in case of one server down =
and the other promote with restart.
The implementation is nearly finished, then begins the stress tests of fail=
ure scenarios.
Rainer
Gesendet: Dienstag, 26. M=E4rz 2013 um 11:55 Uhr
Von: "Steven Bambling" <smbambling at arin.net<mailto:smbambling at arin.net>>
An: "The Pacemaker cluster resource manager" <pacemaker at oss.clusterlabs.org=
<mailto:pacemaker at oss.clusterlabs.org>>
Betreff: Re: [Pacemaker] OCF Resource agent promote question
On Mar 26, 2013, at 6:32 AM, Rainer Brestan <rainer.brestan at gmx.net<x-msg:/=
/211/rainer.brestan at gmx.net>> wrote:
Hi Steve,
when Pacemaker does promotion, it has already selected a specific node to b=
ecome master.
It is far too late in this state to try to update master scores.
But there is another problem with xlog in PostgreSQL.
According to some discussion on PostgreSQL mailing lists, not relevant xlog=
entries dont go into the xlog counter during redo and/or start. This is sp=
ecially true for CHECKPOINT xlog records, where this situation can be easel=
y reproduced.
This can lead to the situation, where the replication is up to date, but th=
e slave shows an lower xlog value.
This issue was solved in 9.2.3, where wal receiver always counts the end of=
applied records.
We are currently testing with 9.2.3. I'm using the functions http://www.da=
tabasesoup.com/2012/10/determining-furthest-ahead-replica.html along with t=
weaking a function to get the replay_lag in bytes to have a more accurate m=
easurement.
There is also a second boring issue. The timeline change is replicated to t=
he slaves, but they do not save it anywhere. In case slave starts up again =
and do not have access to the WAL archive, it cannot start any more. This w=
as also addressed as patch in 9.2 branch, but i havent test if also fixed i=
n 9.2.3.
After talking with one of the Postgres guys it was recommended that we look=
at an alternative solution to the built in trigger file that will make the=
master jump to a new timeline. We are in place moving the recovery.conf t=
o recovery.done via the resource agent and then restarting the the postgres=
ql service on the "new" master so that it maintains the original timeline t=
hat the slaves will recognize.
For data replication, no matter if PostgreSQL or any other database, you ha=
ve always two choices of work.
- Data consistency is the top most priority. Dont go in operation, unless e=
verything fine.
- Availability is the top most priority. Always try to have at least one ru=
nning instance, even if data might not be latest.
The current pgsql RA does quite a good job for the first choice.
It currently has some limitations.
- After switchover, no matter of manual/automatic, it needs some work from =
maintenance personnel.
- Some failure scenarios of fault series lead to a non existing master with=
out manual work.
- Geo-redundant replication with multi-site cluster ticket system (booth) d=
oes not work.
- If availability or unattended work is the priority, it cannot be used out=
of the box.
But it has a very good structure to be extended for other needs.
And this is what i currently implement.
Extend the RA to support both choices of work and prepare it for a multi-si=
te cluster ticket system.
Would you be willing to share your extended RA? Also do you run a cluster =
with more then 2 nodes ?
v/r
STEVE
Regards, Rainer
Gesendet: Dienstag, 26. M=E4rz 2013 um 00:01 Uhr
Von: "Andreas Kurz" <andreas at hastexo.com<x-msg://211/andreas@hastexo.com>>
An: pacemaker at oss.clusterlabs.org<x-msg://211/pacemaker@oss.clusterlabs.org=
>
Betreff: Re: [Pacemaker] OCF Resource agent promote question
Hi Steve,
On 2013-03-25 18:44, Steven Bambling wrote:
> All,
>
> I'm trying to work on a OCF resource agent that uses postgresql
> streaming replication. I'm running into a few issues that I hope might
> be answered or at least some pointers given to steer me in the right
> direction.
Why are you not using the existing pgsql RA? It is capable of doing
synchronous and asynchronous replication and it is known to work fine.
Best regards,
Andreas
--
Need help with Pacemaker?
http://www.hastexo.com/now
>
> 1. A quick way of obtaining a list of "Online" nodes in the cluster
> that a resource will be able to migrate to. I've accomplished it with
> some grep and see but its not pretty or fast.
>
> # time pcs status | grep Online | sed -e "s/.*\[\(.*\)\]/\1/" | sed 's/ /=
/'
> p1.example.net<http://p1.example.net/> <http://p1.example.net<http://p1.e=
xample.net/>> p2.example.net<http://p2.example.net/>
> <http://p2.example.net<http://p2.example.net/>>
>
> real0m2.797s
> user0m0.084s
> sys0m0.024s
>
> Once I get a list of active/online nodes in the cluster my thinking was
> to use PSQL to get the current xlog location and lag or each of the
> remaining nodes and compare them. If the node has a greater log
> position and/or less lag it will be given a greater master preference.
>
> 2. How to force a monitor/probe before a promote is run on ALL nodes to
> make sure that the master preference is up to date before
> migrating/failing over the resource.
> - I was thinking that maybe during the promote call it could get the log
> location and lag from each of the nodes via an psql call ( like above)
> and then force the resource to a specific node. Is there a way to do
> this and does it sound like a sane idea ?
>
>
> The start of my RA is located here suggestions and comments 100%
> welcome https://github.com/smbambling/pgsqlsr/blob/master/pgsqlsr
>
> v/r
>
> STEVE
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org<x-msg://211/Pacemak=
er at oss.clusterlabs.org>
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org<http://www.clusterlabs.org/>
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org<http://bugs.clusterlabs.org/>
>
_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org<x-msg://211/Pacemaker=
@oss.clusterlabs.org>
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org<http://www.clusterlabs.org/>
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org<http://bugs.clusterlabs.org/>
_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org<x-msg://211/Pacemaker=
@oss.clusterlabs.org>
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org<http://www.clusterlabs.org/>
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org<http://bugs.clusterlabs.org/>
_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org<mailto:Pacemaker at oss.=
clusterlabs.org>
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker at oss.clusterlabs.org<mailto:Pacemaker at oss.=
clusterlabs.org>
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
--_000_7CE146047402A34B9E7F0756705B48398CF4ECCACHAXCH01corpari_
Content-Type: text/html; charset="iso-8859-1"
Content-ID: <74DB22831F7D8F4ABCCFC78C452D88E5 at corp.arin.net>
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; ">
<div>I'm reading the additions that you added to the pgsql resource agent t=
o allow for streaming replication in Postgres 9.1+. I'm trying to=
determine if your resource agent will compensate if node promoted ( new ma=
ster ) does not have the newest data. </div>
<div><br>
</div>
<div>From the looks of the pgsql_pre_promote function it seems that it will=
just fail other replicas (slaves) that have newer data, but will continue =
with the promotion of the new master even though it does not have the lates=
t data. </div>
<div><br>
</div>
<div>If this is correct is there a way to force the promotion of the node w=
ith the newest data?</div>
<div><br>
</div>
<div>v/r</div>
<div><br>
</div>
<div>STEVE</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>On Mar 26, 2013, at 8:19 AM, Steven Bambling <<a href=3D"mailto:smb=
ambling at arin.net">smbambling at arin.net</a>> wrote:</div>
<br class=3D"Apple-interchange-newline">
<blockquote type=3D"cite">
<div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line=
-break: after-white-space; ">
Excellent thanks so much for the clarification. I'll drop this new RA=
in and see if I can get things working.
<div><br>
</div>
<div>STEVE</div>
<div><br>
</div>
<div><br>
<div>
<div>On Mar 26, 2013, at 7:38 AM, Rainer Brestan <<a href=3D"mailto:rain=
er.brestan at gmx.net">rainer.brestan at gmx.net</a>></div>
<div> wrote:</div>
<br class=3D"Apple-interchange-newline">
<blockquote type=3D"cite">
<div>
<div style=3D"font-family: Verdana;font-size: 12.0px;">
<div>
<div>Hi Steve,</div>
<div>pgsql RA does the same, it compares the last_xlog_replay_location of a=
ll nodes for master promotion.</div>
<div>Doing a promote as a restart instead of promote command to conserve ti=
meline id is also on configurable option (restart_on_promote<span class=3D"=
nv">)
</span>of the current RA.</div>
<div>And the RA is definitely capable of having more than two instances. It=
goes through the parameter node_list and doing its actions for every membe=
r in the node list.</div>
<div>Originally it might be planned only to have only one slave, but the cu=
rrent implementation does not have this limitation. It has code for sync re=
plication of more than two nodes, when some of them fall back into async to=
not promote them.</div>
<div> </div>
<div>Of course, i will share the extension with the community, when they ar=
e ready for use. And the feature of having more than two instances is not r=
emoved. I am not running more than two instances on one site, current usage=
is to have two instances on one
site and having two sites and manage master by booth. But it also under di=
scussion to have more than two instances on one site, just to have no avail=
ability interruption in case of one server down and the other promote with =
restart.</div>
<div>The implementation is nearly finished, then begins the stress tests of=
failure scenarios.</div>
<div> </div>
<div>Rainer</div>
<div name=3D"quote" style=3D"margin: 10px 5px 5px 10px; padding: 10px 0px 1=
0px 10px; border-left-width: 2px; border-left-style: solid; border-left-col=
or: rgb(195, 217, 229); word-wrap: break-word; -webkit-nbsp-mode: space; -w=
ebkit-line-break: after-white-space; position: static; z-index: auto; ">
<div style=3D"margin:0 0 10px 0;"><b>Gesendet:</b> Dienstag, 26. M=E4r=
z 2013 um 11:55 Uhr<br>
<b>Von:</b> "Steven Bambling" <<a href=3D"mailto:smbambli=
ng at arin.net">smbambling at arin.net</a>><br>
<b>An:</b> "The Pacemaker cluster resource manager" <<a h=
ref=3D"mailto:pacemaker at oss.clusterlabs.org">pacemaker at oss.clusterlabs.org<=
/a>><br>
<b>Betreff:</b> Re: [Pacemaker] OCF Resource agent promote question</d=
iv>
<div name=3D"quoted-content">
<div>
<div>
<div>On Mar 26, 2013, at 6:32 AM, Rainer Brestan <<a href=3D"x-msg://211=
/rainer.brestan at gmx.net" target=3D"_parent">rainer.brestan at gmx.net</a>> =
wrote:</div>
<blockquote>
<div>
<div style=3D"font-family: Verdana;font-size: 12.0px;">
<div>
<div>Hi Steve,</div>
<div>when Pacemaker does promotion, it has already selected a specific node=
to become master.</div>
<div>It is far too late in this state to try to update master scores.</div>
<div> </div>
<div>But there is another problem with xlog in PostgreSQL.</div>
<div> </div>
<div>According to some discussion on PostgreSQL mailing lists, not relevant=
xlog entries dont go into the xlog counter during redo and/or start. This =
is specially true for CHECKPOINT xlog records, where this situation can be =
easely reproduced.</div>
<div>This can lead to the situation, where the replication is up to date, b=
ut the slave shows an lower xlog value.</div>
<div>This issue was solved in 9.2.3, where wal receiver always counts the e=
nd of applied records.</div>
</div>
</div>
</div>
</blockquote>
<div> </div>
We are currently testing with 9.2.3. I'm using the functions <a =
href=3D"http://www.databasesoup.com/2012/10/determining-furthest-ahead-repl=
ica.html" target=3D"_blank">http://www.databasesoup.com/2012/10/determining=
-furthest-ahead-replica.html</a> along with tweaking
a function to get the replay_lag in bytes to have a more accurate measurem=
ent.
<blockquote>
<div>
<div style=3D"font-family: Verdana;font-size: 12.0px;">
<div>
<div> </div>
<div>There is also a second boring issue. The timeline change is replicated=
to the slaves, but they do not save it anywhere. In case slave starts up a=
gain and do not have access to the WAL archive, it cannot start any more. T=
his was also addressed as patch
in 9.2 branch, but i havent test if also fixed in 9.2.3.</div>
</div>
</div>
</div>
</blockquote>
<br>
After talking with one of the Postgres guys it was recommended that we look=
at an alternative solution to the built in trigger file that will make the=
master jump to a new timeline. We are in place moving the recovery.c=
onf to recovery.done via the resource
agent and then restarting the the postgresql service on the "new"=
; master so that it maintains the original timeline that the slaves will re=
cognize.
<blockquote>
<div>
<div style=3D"font-family: Verdana;font-size: 12.0px;">
<div>
<div> </div>
<div>For data replication, no matter if PostgreSQL or any other database, y=
ou have always two choices of work.</div>
<div>- Data consistency is the top most priority. Dont go in operation, unl=
ess everything fine.</div>
<div>- Availability is the top most priority. Always try to have at least o=
ne running instance, even if data might not be latest.</div>
<div> </div>
<div>The current pgsql RA does quite a good job for the first choice.</div>
<div> </div>
<div>It currently has some limitations.</div>
<div>- After switchover, no matter of manual/automatic, it needs some work =
from maintenance personnel.</div>
<div>- Some failure scenarios of fault series lead to a non existing master=
without manual work.</div>
<div>- Geo-redundant replication with multi-site cluster ticket system (boo=
th) does not work.</div>
<div>- If availability or unattended work is the priority, it cannot be use=
d out of the box.</div>
<div> </div>
<div>But it has a very good structure to be extended for other needs.</div>
<div> </div>
<div>And this is what i currently implement.</div>
<div>Extend the RA to support both choices of work and prepare it for a mul=
ti-site cluster ticket system.</div>
</div>
</div>
</div>
</blockquote>
<div> </div>
Would you be willing to share your extended RA? Also do you run a clu=
ster with more then 2 nodes ?</div>
<div> </div>
<div>v/r</div>
<div> </div>
<div>STEVE</div>
<div> </div>
<div>
<blockquote>
<div>
<div style=3D"font-family: Verdana;font-size: 12.0px;">
<div>
<div> </div>
<div>Regards, Rainer</div>
<div style=3D"margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0 10.0px 10=
.0px;border-left: 2.0px solid rgb(195,217,229);">
<div style=3D"margin: 0 0 10.0px 0;"><b>Gesendet:</b> Dienstag, 26. M=
=E4rz 2013 um 00:01 Uhr<br>
<b>Von:</b> "Andreas Kurz" <<a href=3D"x-msg://211/andrea=
s at hastexo.com" target=3D"_parent">andreas at hastexo.com</a>><br>
<b>An:</b> <a href=3D"x-msg://211/pacemaker@oss.clusterlabs.org" targe=
t=3D"_parent">pacemaker at oss.clusterlabs.org</a><br>
<b>Betreff:</b> Re: [Pacemaker] OCF Resource agent promote question</d=
iv>
<div>Hi Steve,<br>
<br>
On 2013-03-25 18:44, Steven Bambling wrote:<br>
> All,<br>
><br>
> I'm trying to work on a OCF resource agent that uses postgresql<br>
> streaming replication. I'm running into a few issues that I hope might=
<br>
> be answered or at least some pointers given to steer me in the right<b=
r>
> direction.<br>
<br>
Why are you not using the existing pgsql RA? It is capable of doing<br>
synchronous and asynchronous replication and it is known to work fine.<br>
<br>
Best regards,<br>
Andreas<br>
<br>
--<br>
Need help with Pacemaker?<br>
<a href=3D"http://www.hastexo.com/now" target=3D"_blank">http://www.hastexo=
.com/now</a><br>
<br>
<br>
><br>
> 1. A quick way of obtaining a list of "Online" nodes in the =
cluster<br>
> that a resource will be able to migrate to. I've accomplished it with<=
br>
> some grep and see but its not pretty or fast.<br>
><br>
> # time pcs status | grep Online | sed -e "s/.*\[\(.*\)\]/\1/"=
; | sed 's/ //'<br>
> <a href=3D"http://p1.example.net/" target=3D"_blank">p1.example.net</a=
> <<a href=3D"http://p1.example.net/" target=3D"_blank">http://p1.exampl=
e.net</a>>
<a href=3D"http://p2.example.net/" target=3D"_blank">p2.example.net</a><br>
> <<a href=3D"http://p2.example.net/" target=3D"_blank">http://p2.exa=
mple.net</a>><br>
><br>
> real0m2.797s<br>
> user0m0.084s<br>
> sys0m0.024s<br>
><br>
> Once I get a list of active/online nodes in the cluster my thinking wa=
s<br>
> to use PSQL to get the current xlog location and lag or each of the<br=
>
> remaining nodes and compare them. If the node has a greater log<br>
> position and/or less lag it will be given a greater master preference.=
<br>
><br>
> 2. How to force a monitor/probe before a promote is run on ALL nodes t=
o<br>
> make sure that the master preference is up to date before<br>
> migrating/failing over the resource.<br>
> - I was thinking that maybe during the promote call it could get the l=
og<br>
> location and lag from each of the nodes via an psql call ( like above)=
<br>
> and then force the resource to a specific node. Is there a way to do<b=
r>
> this and does it sound like a sane idea ?<br>
><br>
><br>
> The start of my RA is located here suggestions and comments 100%<br>
> welcome <a href=3D"https://github.com/smbambling/pgsqlsr/blob/master/p=
gsqlsr" target=3D"_blank">
https://github.com/smbambling/pgsqlsr/blob/master/pgsqlsr</a><br>
><br>
> v/r<br>
><br>
> STEVE<br>
><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href=3D"x-msg://211/Pacemaker@oss.clusterla=
bs.org" target=3D"_parent">
Pacemaker at oss.clusterlabs.org</a><br>
> <a href=3D"http://oss.clusterlabs.org/mailman/listinfo/pacemaker" targ=
et=3D"_blank">
http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href=3D"http://www.clusterlabs.org/" target=3D"_blank=
">http://www.clusterlabs.org</a><br>
> Getting started: <a href=3D"http://www.clusterlabs.org/doc/Cluster_fro=
m_Scratch.pdf" target=3D"_blank">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href=3D"http://bugs.clusterlabs.org/" target=3D"_blank">http:=
//bugs.clusterlabs.org</a><br>
><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href=3D"x-msg://211/Pacemaker@oss.clusterlabs.or=
g" target=3D"_parent">
Pacemaker at oss.clusterlabs.org</a><br>
<a href=3D"http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target=3D=
"_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href=3D"http://www.clusterlabs.org/" target=3D"_blank">htt=
p://www.clusterlabs.org</a><br>
Getting started: <a href=3D"http://www.clusterlabs.org/doc/Cluster_from_Scr=
atch.pdf" target=3D"_blank">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href=3D"http://bugs.clusterlabs.org/" target=3D"_blank">http://bug=
s.clusterlabs.org</a></div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Pacemaker mailing list: <a href=3D"x-msg://211/Pacemaker@oss.clusterlabs.or=
g" target=3D"_parent">
Pacemaker at oss.clusterlabs.org</a><br>
<a href=3D"http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target=3D=
"_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href=3D"http://www.clusterlabs.org/">http://www.clusterlab=
s.org</a><br>
Getting started: <a href=3D"http://www.clusterlabs.org/doc/Cluster_from_Scr=
atch.pdf">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href=3D"http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org<=
/a></blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Pacemaker mailing list: <a href=3D"mailto:Pacemaker at oss.clusterlabs.org">Pa=
cemaker at oss.clusterlabs.org</a><br>
<a href=3D"http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://os=
s.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href=3D"http://www.clusterlabs.org">http://www.clusterlabs=
.org</a><br>
Getting started: <a href=3D"http://www.clusterlabs.org/doc/Cluster_from_Scr=
atch.pdf">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href=3D"http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</=
a><br>
</blockquote>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Pacemaker mailing list: <a href=3D"mailto:Pacemaker at oss.clusterlabs.org">Pa=
cemaker at oss.clusterlabs.org</a><br>
<a href=3D"http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://os=
s.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: http://www.clusterlabs.org<br>
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br=
>
Bugs: http://bugs.clusterlabs.org<br>
</blockquote>
</div>
<br>
</body>
</html>
--_000_7CE146047402A34B9E7F0756705B48398CF4ECCACHAXCH01corpari_--
More information about the Pacemaker
mailing list