[Pacemaker] Postgresql streaming replication failover - RA needed
Takatoshi MATSUO
matsuo.tak at gmail.com
Sun Nov 27 20:10:18 EST 2011
Hi Attila
Primary can not send all wals to HotStandby whether primary is shut
down normally.
These logs validate it.
> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: My Timeline ID and Checkpoint : 14:0000000023000020
> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: psql2 master baseline : 14:0000000023000000
psql1's location was "0000000023000020" when it was demoted.
OTOH psql2's location was "0000000023000000" when it was promoted.
It means that psql1's data was newer than psql2's one at that time.
The gap is 20.
As you said you can start psql1's PostgreSQL manually,
but PostgreSQL can't realize this occurrence.
If you start HotStandby at psql1, data is replicated after 0000000023000020.
It's inconsistency.
Thanks,
Takatoshi MATSUO
2011/11/28 Attila Megyeri <amegyeri at minerva-soft.com>:
> Hi Takatoshi,
>
> I don't think it is inconsistency problem - for me it looks like some RA bug.
> I think so, because postgres starts properly outside pacemaker.
>
> When pacemaker starts node psql1 I see only:
>
> postgresql:0_start_0 (node=psql1, call=9, rc=1, status=complete): unknown error
>
> and the postgres log is empty - so I suppose that it does not even try to start it.
>
> What I tested was:
> - I had a stable cluster, where psql1 was the master, psql2 was the slave
> - I put psql1 into standby mode. ("node psql1 standby") to test failover
> - After a while psql2 became the PRI, which is very good
> - When I put psql1 back online, postgres wouldn't start anymore from pacemaker (unknown error).
>
>
> I tried to start postgres manually from the shell it worked fine, even the monitor was able to see that it became in SYNC (obviously the master/slave group was showing improper state as psql was started outside pacemaker.
>
> I don't think data inconsistency is the case, partially because there are no clients connected, partially because psql starts properly outside pacemaker.
>
> Here is what is relevant from the log:
>
> Nov 27 16:02:50 psql1 pgsql[11021]: DEBUG: PostgreSQL is running as a primary.
> Nov 27 16:02:51 psql1 pgsql[11021]: DEBUG: node=psql2, state=STREAMING, sync_state=SYNC
> Nov 27 16:02:53 psql1 pgsql[11142]: DEBUG: PostgreSQL is running as a primary.
> Nov 27 16:02:53 psql1 pgsql[11142]: DEBUG: node=psql2, state=STREAMING, sync_state=SYNC
> Nov 27 16:02:55 psql1 pgsql[11272]: DEBUG: PostgreSQL is running as a primary.
> Nov 27 16:02:55 psql1 pgsql[11272]: DEBUG: node=psql2, state=STREAMING, sync_state=SYNC
> Nov 27 16:02:57 psql1 pgsql[11368]: DEBUG: PostgreSQL is running as a primary.
> Nov 27 16:02:57 psql1 pgsql[11368]: DEBUG: node=psql2, state=STREAMING, sync_state=SYNC
> Nov 27 16:03:00 psql1 pgsql[11463]: DEBUG: PostgreSQL is running as a primary.
> Nov 27 16:03:00 psql1 pgsql[11463]: DEBUG: node=psql2, state=STREAMING, sync_state=SYNC
> Nov 27 16:03:00 psql1 pgsql[11556]: DEBUG: notify: pre for demote
> Nov 27 16:03:00 psql1 pgsql[11590]: INFO: Stopping PostgreSQL on demote.
> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: waiting for server to shut down..... done server stopped
> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: Removing /var/lib/pgsql/PGSQL.lock.
> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: PostgreSQL is down
> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: Changing pgsql-status on psql1 : PRI->STOP.
> Nov 27 16:03:02 psql1 pgsql[11590]: DEBUG: Created recovery.conf. host=10.12.1.28, user=postgres
> Nov 27 16:03:02 psql1 pgsql[11590]: INFO: Setup all nodes as an async.
> Nov 27 16:03:02 psql1 pgsql[11732]: DEBUG: notify: post for demote
> Nov 27 16:03:02 psql1 pgsql[11732]: DEBUG: post-demote called. Demote uname is psql1
> Nov 27 16:03:02 psql1 pgsql[11732]: INFO: My Timeline ID and Checkpoint : 14:0000000023000020
> Nov 27 16:03:02 psql1 pgsql[11732]: WARNING: Can't get psql2 master baseline. Waiting...
> Nov 27 16:03:03 psql1 pgsql[11732]: INFO: psql2 master baseline : 14:0000000023000000
> Nov 27 16:03:03 psql1 pgsql[11732]: ERROR: My data is inconsistent.
> Nov 27 16:03:03 psql1 pgsql[11867]: DEBUG: notify: pre for stop
> Nov 27 16:03:03 psql1 pgsql[11969]: INFO: PostgreSQL is already stopped.
> Nov 27 16:03:12 psql1 pgsql[12053]: INFO: Don't check /var/lib/postgresql/9.1/main during probe
> Nov 27 16:03:12 psql1 pgsql[12053]: INFO: PostgreSQL is down
> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: Changing pgsql-status on psql1 : ->STOP.
> Nov 27 16:03:27 psql1 pgsql[12204]: DEBUG: Created recovery.conf. host=10.12.1.28, user=postgres
> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: Setup all nodes as an async.
> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: My Timeline ID and Checkpoint : 14:0000000023000020
> Nov 27 16:03:27 psql1 pgsql[12204]: INFO: psql2 master baseline : 14:0000000023000000
> Nov 27 16:03:27 psql1 pgsql[12204]: ERROR: My data is inconsistent.
> Nov 27 16:03:27 psql1 pgsql[12339]: DEBUG: notify: post for start
> Nov 27 16:03:27 psql1 pgsql[12373]: DEBUG: notify: pre for stop
> Nov 27 16:03:27 psql1 pgsql[12407]: INFO: PostgreSQL is already stopped.
>
>
> Thanks,
>
> Attila
>
>
> -----Original Message-----
> From: Takatoshi MATSUO [mailto:matsuo.tak at gmail.com]
> Sent: 2011. november 27. 11:07
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Postgresql streaming replication failover - RA needed
>
> Hi Attila
>
> 2011/11/27 Attila Megyeri <amegyeri at minerva-soft.com>:
>> Hi Takatoshi,
>>
>> You were right, changing the shell to bash resolved the problem.
>> The cluster now started in sync mode - thank you very much.
>
> You're very welcome.
>
>> I will be testing it in the next couple of days. I did just a very quick test - it seems that psql
>> master failed over to psql2 properly, but when I tried to move it back to psql1 there was some
>> problems starting psql on node 1.
>
> If master(psql1) is failed, its data may be inconsistency.
> A PostgreSQL developer says that it's a feature.
> Therefore my RA prevent it from starting automatically if data is inconsistency.
> Please backup psql2' data and restore it to psql1, and remove
> /var/lib/pgsql/PGSQL.lock file
> before clearing failcount.
>
> I use rsync to backup and restore in the following way.
> -----
> # psql -h 192.168.2.114 -U postgres -c "SELECT pg_start_backup('label')"
> # rsync -avr --delete --exclude=postmaster.pid
> 192.168.2.114:/var/lib/pgsql/9.1/data/ /var/lib/pgsql/9.1/data/
> # psql -h 192.168.2.114 -U postgres -c "SELECT pg_stop_backup()"
> -----
>
>
> BTW I fixed some bugs 2 days ago.
> Please use the newest version.
>
> Thanks,
> Takatoshi MATSUO
>
>
>>
>> Does it work fine for you in both directions?
>>
>> Thank you very much.
>>
>> Have a nice weekend,
>>
>> Attila
>>
>>
>>
>> -----Original Message-----
>> From: Takatoshi MATSUO [mailto:matsuo.tak at gmail.com]
>> Sent: 2011. november 27. 6:12
>> To: The Pacemaker cluster resource manager
>> Subject: Re: [Pacemaker] Postgresql streaming replication failover - RA needed
>>
>> Hi Attila
>>
>> 2011/11/27 Attila Megyeri <amegyeri at minerva-soft.com>:
>>> Hi Takatoshi,
>>>
>>> Thank you for coming back to me so quickly.
>>>
>>> In the /var/lib/pgsql there are the following files:
>>>
>>> PSQL1:
>>> =====
>>> root at psql1:/var/lib/pgsql# ls -la
>>> total 16
>>> drwxr-xr-x 2 postgres postgres 4096 Nov 26 18:04 .
>>> drwxr-xr-x 35 root root 4096 Nov 25 22:21 ..
>>> -rw-r--r-- 1 postgres postgres 1 Nov 26 00:17 rep_mode.conf
>>> -rw-r--r-- 1 root root 49 Nov 26 18:04 xlog_note.0
>>>
>>> root at psql1:/var/lib/pgsql# cat xlog_note.0 -e psql1 0000000019000000
>>> psql2 0000000019000000
>>> root at psql1:/var/lib/pgsql#
>>>
>>> PSQL2:
>>> =======
>>> root at psql2:/var/lib/pgsql# ls -la
>>> total 16
>>> drwxr-xr-x 2 postgres postgres 4096 Nov 26 18:05 .
>>> drwxr-xr-x 33 root root 4096 Nov 26 00:10 ..
>>> -rw-r--r-- 1 postgres postgres 1 Nov 26 00:24 rep_mode.conf
>>> -rw-r--r-- 1 root root 49 Nov 26 18:05 xlog_note.0
>>> root at psql2:/var/lib/pgsql# cat xlog_note.0 -e psql1 0000000019000000
>>> psql2 0000000019000000
>>> root at psql2:/var/lib/pgsql#
>>
>> It seems that dash's bultin echo command is used because echo with "-e" option dose not function.
>>
>> Perhaps my RA also depends on bash.
>> Can you use a bash instead of a dash?
>>
>>> BTW, postgres is installed under /var/lib/postgresql , but I noticed that some parts of the RA are referring to the /var/lib/pgsql directory, so I created that directory and i keep some of the files there.
>>
>> It's no ploblem.
>> If you want to change this path, please specify it using "tmpdir" parameter.
>>
>> Regards,
>> Takatoshi MATSUO
>>
>>>
>>> Thanks,
>>> Attila
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Takatoshi MATSUO [mailto:matsuo.tak at gmail.com]
>>> Sent: 2011. november 26. 18:27
>>> To: The Pacemaker cluster resource manager
>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover -
>>> RA needed
>>>
>>> Hi Attila
>>>
>>> 1. Are there /var/lib/pgsql/xlog_note.0 , xlog_note.1, xlog_note.2 .... files?
>>> These files are created while checking a xlog location on monitor.
>>>
>>> 2. Do these files include lines as below?
>>> ---------
>>> pgsql1 0000000019000000
>>> pgsql2 0000000019000000
>>> ---------
>>>
>>> Regards.
>>> Takatoshi MATSUO
>>>
>>>
>>> 2011年11月26日22:44 Attila Megyeri <amegyeri at minerva-soft.com>:
>>>> Hi Yoshiharu, Takatoshi,
>>>>
>>>> Spent another day, without success. :(
>>>>
>>>> I started from scratch and synchronous replications works nicely when nodes are started outside pacemaker.
>>>> My PostgreSQL version is 9.1.1.
>>>>
>>>> When I start from pacemaker, after a while it gets into the following state:
>>>>
>>>> Online: [ psql1 psql2 ]
>>>>
>>>> Master/Slave Set: msPostgresql [postgresql]
>>>> Slaves: [ psql1 psql2 ]
>>>> Clone Set: clnPingCheck [pingCheck]
>>>> Started: [ psql1 psql2 ]
>>>>
>>>> Node Attributes:
>>>> * Node psql1:
>>>> + default_ping_set : 100
>>>> + master-postgresql:0 : -INFINITY
>>>> + pgsql-status : HS:alone
>>>> + pgsql-xlog-loc : 0000000019000000
>>>> * Node psql2:
>>>> + default_ping_set : 100
>>>> + master-postgresql:1 : -INFINITY
>>>> + pgsql-status : HS:alone
>>>> + pgsql-xlog-loc : 0000000019000000
>>>>
>>>>
>>>> The psql status queries return the following:
>>>>
>>>> PSQL1
>>>> ======
>>>> postgres at psql1:/root$ psql -c "select application_name,upper(state),upper(sync_state) from pg_stat_replication"
>>>> application_name | upper | upper
>>>> ------------------+-------+-------
>>>> (0 rows)
>>>>
>>>> postgres at psql1:/root$ psql -Atc "select pg_last_xlog_replay_location(),pg_last_xlog_receive_location()"
>>>> 0/19000020|0/19000000
>>>>
>>>> PSQL2
>>>> ======
>>>> postgres at psql2:~$ psql -c "select application_name,upper(state),upper(sync_state) from pg_stat_replication"
>>>> application_name | upper | upper
>>>> ------------------+-------+-------
>>>> (0 rows)
>>>>
>>>> postgres at psql2:~$ psql -Atc "select pg_last_xlog_replay_location(),pg_last_xlog_receive_location()"
>>>> 0/19000000|0/19000000
>>>>
>>>>
>>>> Neither server can connect (obviously) to the master, as the vip_repl Is not brought up.
>>>>
>>>>
>>>> Could you help me understand WHAT is the action/state/event that sould promote one of the nodes? I see that pacemaker monitors the servers every X seconds, but nothing else happens.
>>>>
>>>> In the log (limited to pgsql) the following sequence is repeated
>>>> forewer
>>>>
>>>> Nov 26 13:36:19 psql1 pgsql[19829]: INFO: Master is not exist.
>>>> Nov 26 13:36:19 psql1 pgsql[19829]: DEBUG: Checking right of master.
>>>> Nov 26 13:36:19 psql1 pgsql[19829]: INFO: My data status=.
>>>> Nov 26 13:36:19 psql1 pgsql[19829]: INFO: psql1 xlog location :
>>>> 0000000019000000 Nov 26 13:36:19 psql1 pgsql[19829]: INFO: psql2 xlog
>>>> location : 0000000019000000 Nov 26 13:36:26 psql1 pgsql[19993]: DEBUG: PostgreSQL is running as a hot standby.
>>>> Nov 26 13:36:26 psql1 pgsql[19993]: INFO: Master is not exist.
>>>> Nov 26 13:36:26 psql1 pgsql[19993]: DEBUG: Checking right of master.
>>>> Nov 26 13:36:26 psql1 pgsql[19993]: INFO: My data status=.
>>>> Nov 26 13:36:26 psql1 pgsql[19993]: INFO: psql1 xlog location :
>>>> 0000000019000000 Nov 26 13:36:26 psql1 pgsql[19993]: INFO: psql2 xlog
>>>> location : 0000000019000000 Nov 26 13:36:33 psql1 pgsql[20176]: DEBUG: PostgreSQL is running as a hot standby.
>>>> Nov 26 13:36:33 psql1 pgsql[20176]: INFO: Master is not exist.
>>>> Nov 26 13:36:33 psql1 pgsql[20176]: DEBUG: Checking right of master.
>>>> Nov 26 13:36:33 psql1 pgsql[20176]: INFO: My data status=.
>>>> Nov 26 13:36:33 psql1 pgsql[20176]: INFO: psql1 xlog location :
>>>> 0000000019000000 Nov 26 13:36:33 psql1 pgsql[20176]: INFO: psql2 xlog
>>>> location : 0000000019000000 Nov 26 13:36:41 psql1 pgsql[20343]: DEBUG: PostgreSQL is running as a hot standby.
>>>>
>>>>
>>>> Any help is appreciated!
>>>>
>>>> Regards,
>>>> Attila
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Yoshiharu Mori [mailto:y-mori at sraoss.co.jp]
>>>> Sent: 2011. november 25. 14:17
>>>> To: The Pacemaker cluster resource manager
>>>> Cc: Attila Megyeri
>>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover -
>>>> RA needed
>>>>
>>>> Hi Attila
>>>>
>>>>> A quick snippet from the corosync.log
>>>>>
>>>>> Nov 23 05:43:05 psql1 pgsql[2845]: DEBUG: Checking right of master.
>>>>> Nov 23 05:43:05 psql1 pgsql[2845]: INFO: My data status=.
>>>>> Nov 23 05:43:05 psql1 pgsql[2845]: INFO: psql1 xlog location :
>>>>> 000000000D000000 Nov 23 05:43:05 psql1 pgsql[2845]: INFO: psql2 xlog
>>>>> location : 0000000008000000
>>>>>
>>>>> As you see, the "my data status" returns an empty string.
>>>>
>>>> My log is same. but it works.
>>>>
>>>> Nov 18 19:28:26 osspc24-1 pgsql[17350]: INFO: Master is not exist.
>>>> Nov 18 19:28:26 osspc24-1 pgsql[17350]: INFO: Checking right of master.
>>>> Nov 18 19:28:19 osspc24-1 pgsql[17138]: INFO: My data status=.
>>>> Nov 18 19:28:19 osspc24-1 pgsql[17138]: INFO: pm01 xlog location :
>>>> 0000000005000020 Nov 18 19:28:19 osspc24-1 pgsql[17138]: INFO: pm02
>>>> xlog location : 0000000005000000
>>>>
>>>> In my log, the following logs are outputted and started after checking xlog location(3 times).
>>>>
>>>> Nov 18 19:29:39 osspc24-1 pgsql[18720]: INFO: I have a master right.
>>>>
>>>> Please show us more corosync.log.
>>>>
>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Attila Megyeri [mailto:amegyeri at minerva-soft.com]
>>>>> Sent: 2011. november 25. 9:28
>>>>> To: The Pacemaker cluster resource manager
>>>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover -
>>>>> RA needed
>>>>>
>>>>> Hi Takatoshi,
>>>>>
>>>>> I have restored the PSQL to run without corosync so I cannot send you the crm_mon output now.
>>>>>
>>>>> What I can tell for sure:
>>>>> - RA never promoted any of the nodes, no matter what the status was. It also did not promote the node, when it was the only one.
>>>>> - I believe the issue is in the comparison of the xlogs. How could I troubleshoot that? I see from the logs that crm NEVER tried to invoke pgsql with "promote"
>>>>> - I tried previously the crm_mon -A option, but there was never a "
>>>>> pgsql-data-status" attribute. The other attribs were there,
>>>>> including the HS:alone
>>>>> - In the corosync log the only relevant RA message I see is " Master is not exist. " I never saw a message like "My data is out-of-date"
>>>>>
>>>>> Thank you!
>>>>>
>>>>> Attila
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Takatoshi MATSUO [mailto:matsuo.tak at gmail.com]
>>>>> Sent: 2011. november 25. 8:56
>>>>> To: The Pacemaker cluster resource manager
>>>>> Subject: Re: [Pacemaker] Postgresql streaming replication failover -
>>>>> RA needed
>>>>>
>>>>> Hi Attila
>>>>>
>>>>> 2011/11/24 Attila Megyeri <amegyeri at minerva-soft.com>:
>>>>> > Hi Takatoshi, All,
>>>>> >
>>>>> > Thanks for your reply.
>>>>> > I see that you have invested significant effort in the development of the RA. I spent the last day trying to set up the RA, but without much success.
>>>>> >
>>>>> > My infrastructure is very similar to yours, except for the fact that currently I am testing with a single network adapter.
>>>>> >
>>>>> > Replication works nicely when I start the databases manually, not using corosync.
>>>>> >
>>>>> > When I try to start using corosync,I see that the ping resources start normally, but the msPostgresql starts on both nodes in slave mode, and I see "HS:alone"
>>>>>
>>>>> To see "HS:alone" is normal.
>>>>> And RA compares xlog locations and promote the postgresql having new data.
>>>>>
>>>>> > In the Wiki you state, the if I start on a signle node only, PSQL should start in Master mode (PRI), but this is not the case.
>>>>>
>>>>> If the data is old, the node can't be master.
>>>>> To be master needs pgsql-data-status="LATEST" or "STREAMING|SYNC".
>>>>> Plese check it using "crm_mon -A".
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> And to become a master from stopped takes a few minutes because the RA compares xlog location on monitor.
>>>>>
>>>>>
>>>>> > The recovery.conf file is created immediately, and from the logs I see no attempt at all to promote the node.
>>>>> > In the postgres logs I see that node1, which is supposed to be a master, tries to connect to the vip-rep IP address, which is NOT brought up, because it depends on the Master role...
>>>>> >
>>>>> > Do you have any idea?
>>>>>
>>>>> Please check HA log.
>>>>> My RA outputs "My data is out-of-date. status=********" to log if the data is old.
>>>>>
>>>>> Regards,
>>>>> Takatoshi MATSUO
>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>>
>>>> --
>>>> Yoshiharu Mori <y-mori at sraoss.co.jp>
>>>> SRA OSS, Inc Japan http://www.sraoss.co.jp
>>>> TEL: 03-5979-2701
>>>> FAX: 03-5979-2702
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org Getting started:
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
More information about the Pacemaker
mailing list