[Pacemaker] Postgresql streaming replication failover - RA needed

[Takatoshi MATSUO]

Hello Serge

2011/12/13 Serge Dubrouski <sergeyfd at gmail.com>:
>
>
> On Mon, Dec 12, 2011 at 5:32 AM, Takatoshi MATSUO <matsuo.tak at gmail.com>
> wrote:
>>
>> Hello
>>
>> 2011/12/12 Serge Dubrouski <sergeyfd at gmail.com>:
>> >
>> >
>> > On Thu, Dec 8, 2011 at 10:34 PM, Takatoshi MATSUO <matsuo.tak at gmail.com>
>> > wrote:
>> >>
>> >> Hi Attila
>> >>
>> >> 2011/12/8 Attila Megyeri <amegyeri at minerva-soft.com>:
>> >> > Hi Takatoshi,
>> >> >
>> >> > One strange thing I noticed and could probably be improved.
>> >> > When there is data inconsistency, I have the following node
>> >> > properties:
>> >> >
>> >> > * Node psql2:
>> >> >    + default_ping_set                  : 100
>> >> >    + master-postgresql:1               : -INFINITY
>> >> >    + pgsql-data-status                 : DISCONNECT
>> >> >    + pgsql-status                      : HS:alone
>> >> > * Node psql1:
>> >> >    + default_ping_set                  : 100
>> >> >    + master-postgresql:0               : 1000
>> >> >    + master-postgresql:1               : -INFINITY
>> >> >    + pgsql-data-status                 : LATEST
>> >> >    + pgsql-master-baseline             : 58:000000004B000020
>> >> >    + pgsql-status                      : PRI
>> >> >
>> >> > This is fine, and understandable - but I can see this only if I do a
>> >> > crm_mon -A.
>> >> >
>> >> > My problem is, that CRM shows the following:
>> >> >
>> >> > Master/Slave Set: db-ms-psql [postgresql]
>> >> >     Masters: [ psql1 ]
>> >> >     Slaves: [ psql2 ]
>> >> >
>> >> > So if I monitor the system from crm_mon, HAWK or ther tools - I have
>> >> > no
>> >> > indication at all that the slave is running in an inconsistent mode.
>> >> >
>> >> > I would expect the RA to stop the psql2 node in such cases, because:
>> >> > - It is running, but has non-up-to-date data, therefore noone will
>> >> > use
>> >> > it (the slave IP points to the master as well, which is good)
>> >> > - In CRM status eveything looks perfect, even though it is NOT
>> >> > perfect
>> >> > and admin intervention is required.
>> >> >
>> >> >
>> >> > Shouldn't the disconnected PSQL server be stopped instead?
>> >>
>> >> hmm..
>> >> It's not better to stop PGSQL server.
>> >> RA cannot know whether PGSQL is disconnected because of
>> >> data-inconsistent or network-down or
>> >> starting-up and so on.
>> >
>> >
>> > Why does it matter? If the state is degraded and inconsistent and there
>> > is
>> > no way to fix it from inside of the RA, RA should probably stop it.
>>
>> In this case, HS's data may be cosistent but Primary dosen't have enough
>> wals or
>> HS dosen't have enough wal-archives to be replication-mode.
>> Unfortunately this RA dosen't calculate the number of wals.
>
>
> Honestly I don't know how to better handle this. Pacemaker doesn't have a
> concept of degraded node state.

In this case the RA cannot know whether it is degraded or not
for the above reason.

Of course, the RA stops PostgreSQL when it is obviously degraded .

>
>>
>>
>> > Let's say that there is pgpool running in front of the cluster, keeping
>> > an
>> > inconsistent node up would lead to the routing SQL queries to it and
>> > possibly getting wrong results.
>> >
>>
>> It dosen't happen in my sample configuration.
>> vip-slave is up at master when slave is not "HS:sync".
>
>
> So you have a VIP for each slave node?
>

Yes.
If you don't need read-only access,
it is no problem removing vip-slave.

>>
>>
>> >>
>> >>
>> >>
>> >> How about using dummy RA such as vip-slave?
>> >> -------------------------------------------
>> >> primitive runningSlaveOK ocf:heartbeat:Dummy
>> >> .....(snip)
>> >>
>> >> location rsc_location-dummy runningSlaveOK \
>> >>     rule  200: pgsql-status eq "HS:sync"
>> >> -------------------------------------------
>>
>> >
>> > That probably fixes visibility issue. What about notifications on
>> > DISCONNECT
>> > state? How administrator would know that cluster is inconsistent? May be
>> > the
>> > better option in this case would be collocating MailTo resource with
>> > "HS:alone"?
>>
>> Yes, it's good idea if you want to receive notifications.
>>
>>

Regards,
Takatoshi MATSUO