[Pacemaker] why pacemaker does not control the resources

Andrew Beekhof andrew at beekhof.net
Wed Nov 13 22:15:20 UTC 2013 

On 14 Nov 2013, at 6:13 am, Andrey Groshev <greenx at yandex.ru> wrote:

> 
> 
> 13.11.2013, 03:22, "Andrew Beekhof" <andrew at beekhof.net>:
>> On 12 Nov 2013, at 4:42 pm, Andrey Groshev <greenx at yandex.ru> wrote:
>> 
>>>  11.11.2013, 03:44, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>  On 8 Nov 2013, at 7:49 am, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>   Hi, PPL!
>>>>>   I need help. I do not understand... Why has stopped working.
>>>>>   This configuration work on other cluster, but on corosync1.
>>>>> 
>>>>>   So... cluster postgres with master/slave.
>>>>>   Classic config as in wiki.
>>>>>   I build cluster, start, he is working.
>>>>>   Next I kill postgres on Master with 6 signal, as if "disk space left"
>>>>> 
>>>>>   # pkill -6 postgres
>>>>>   # ps axuww|grep postgres
>>>>>   root      9032  0.0  0.1 103236   860 pts/0    S+   00:37   0:00 grep postgres
>>>>> 
>>>>>   PostgreSQL die, But crm_mon shows that the master is still running.
>>>>> 
>>>>>   Last updated: Fri Nov  8 00:42:08 2013
>>>>>   Last change: Fri Nov  8 00:37:05 2013 via crm_attribute on dev-cluster2-node4
>>>>>   Stack: corosync
>>>>>   Current DC: dev-cluster2-node4 (172793107) - partition with quorum
>>>>>   Version: 1.1.10-1.el6-368c726
>>>>>   3 Nodes configured
>>>>>   7 Resources configured
>>>>> 
>>>>>   Node dev-cluster2-node2 (172793105): online
>>>>>          pingCheck       (ocf::pacemaker:ping):  Started
>>>>>          pgsql   (ocf::heartbeat:pgsql): Started
>>>>>   Node dev-cluster2-node3 (172793106): online
>>>>>          pingCheck       (ocf::pacemaker:ping):  Started
>>>>>          pgsql   (ocf::heartbeat:pgsql): Started
>>>>>   Node dev-cluster2-node4 (172793107): online
>>>>>          pgsql   (ocf::heartbeat:pgsql): Master
>>>>>          pingCheck       (ocf::pacemaker:ping):  Started
>>>>>          VirtualIP       (ocf::heartbeat:IPaddr2):       Started
>>>>> 
>>>>>   Node Attributes:
>>>>>   * Node dev-cluster2-node2:
>>>>>      + default_ping_set                  : 100
>>>>>      + master-pgsql                      : -INFINITY
>>>>>      + pgsql-data-status                 : STREAMING|ASYNC
>>>>>      + pgsql-status                      : HS:async
>>>>>   * Node dev-cluster2-node3:
>>>>>      + default_ping_set                  : 100
>>>>>      + master-pgsql                      : -INFINITY
>>>>>      + pgsql-data-status                 : STREAMING|ASYNC
>>>>>      + pgsql-status                      : HS:async
>>>>>   * Node dev-cluster2-node4:
>>>>>      + default_ping_set                  : 100
>>>>>      + master-pgsql                      : 1000
>>>>>      + pgsql-data-status                 : LATEST
>>>>>      + pgsql-master-baseline             : 0000000002000078
>>>>>      + pgsql-status                      : PRI
>>>>> 
>>>>>   Migration summary:
>>>>>   * Node dev-cluster2-node4:
>>>>>   * Node dev-cluster2-node2:
>>>>>   * Node dev-cluster2-node3:
>>>>> 
>>>>>   Tickets:
>>>>> 
>>>>>   CONFIG:
>>>>>   node $id="172793105" dev-cluster2-node2. \
>>>>>          attributes pgsql-data-status="STREAMING|ASYNC" standby="false"
>>>>>   node $id="172793106" dev-cluster2-node3. \
>>>>>          attributes pgsql-data-status="STREAMING|ASYNC" standby="false"
>>>>>   node $id="172793107" dev-cluster2-node4. \
>>>>>          attributes pgsql-data-status="LATEST"
>>>>>   primitive VirtualIP ocf:heartbeat:IPaddr2 \
>>>>>          params ip="10.76.157.194" \
>>>>>          op start interval="0" timeout="60s" on-fail="stop" \
>>>>>          op monitor interval="10s" timeout="60s" on-fail="restart" \
>>>>>          op stop interval="0" timeout="60s" on-fail="block"
>>>>>   primitive pgsql ocf:heartbeat:pgsql \
>>>>>          params pgctl="/usr/pgsql-9.1/bin/pg_ctl" psql="/usr/pgsql-9.1/bin/psql" pgdata="/var/lib/pgsql/9.1/data" tmpdir="/tmp/pg" start_opt="-p 5432" logfile="/var/lib/pgsql/9.1//pgstartup.log" rep_mode="async" node_list=" dev-cluster2-node2. dev-cluster2-node3. dev-cluster2-node4. " restore_command="gzip -cd /var/backup/pitr/dev-cluster2-master#5432/xlog/%f.gz > %p" primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 keepalives_count=5" master_ip="10.76.157.194" \
>>>>>          op start interval="0" timeout="60s" on-fail="restart" \
>>>>>          op monitor interval="5s" timeout="61s" on-fail="restart" \
>>>>>          op monitor interval="1s" role="Master" timeout="62s" on-fail="restart" \
>>>>>          op promote interval="0" timeout="63s" on-fail="restart" \
>>>>>          op demote interval="0" timeout="64s" on-fail="stop" \
>>>>>          op stop interval="0" timeout="65s" on-fail="block" \
>>>>>          op notify interval="0" timeout="66s"
>>>>>   primitive pingCheck ocf:pacemaker:ping \
>>>>>          params name="default_ping_set" host_list="10.76.156.1" multiplier="100" \
>>>>>          op start interval="0" timeout="60s" on-fail="restart" \
>>>>>          op monitor interval="10s" timeout="60s" on-fail="restart" \
>>>>>          op stop interval="0" timeout="60s" on-fail="ignore"
>>>>>   ms msPostgresql pgsql \
>>>>>          meta master-max="1" master-node-max="1" clone-node-max="1" notify="true" target-role="Master" clone-max="3"
>>>>>   clone clnPingCheck pingCheck \
>>>>>          meta clone-max="3"
>>>>>   location l0_DontRunPgIfNotPingGW msPostgresql \
>>>>>          rule $id="l0_DontRunPgIfNotPingGW-rule" -inf: not_defined default_ping_set or default_ping_set lt 100
>>>>>   colocation r0_StartPgIfPingGW inf: msPostgresql clnPingCheck
>>>>>   colocation r1_MastersGroup inf: VirtualIP msPostgresql:Master
>>>>>   order rsc_order-1 0: clnPingCheck msPostgresql
>>>>>   order rsc_order-2 0: msPostgresql:promote VirtualIP:start symmetrical=false
>>>>>   order rsc_order-3 0: msPostgresql:demote VirtualIP:stop symmetrical=false
>>>>>   property $id="cib-bootstrap-options" \
>>>>>          dc-version="1.1.10-1.el6-368c726" \
>>>>>          cluster-infrastructure="corosync" \
>>>>>          stonith-enabled="false" \
>>>>>          no-quorum-policy="stop"
>>>>>   rsc_defaults $id="rsc-options" \
>>>>>          resource-stickiness="INFINITY" \
>>>>>          migration-threshold="1"
>>>>> 
>>>>>   Tell me where to look - why does pacemaker not work?
>>>>  You might want to follow some of the steps at:
>>>> 
>>>>     http://blog.clusterlabs.org/blog/2013/debugging-pacemaker/
>>>> 
>>>>  under the heading "Resource-level failures".
>>>  Yes. Thank you.
>>>  I've seen this article and now I study it in more detail.
>>>  A lot of information in the logs, so it is difficult to determine where the error is, and where the consequence of error.
>>>  Now I'm trying to figure it out.
>>> 
>>>  BUT...
>>>  While I can say with certainty that the RA with monitor in the MS(pgsql) is called ONLY on the node on which the last was launched PACEMAKER.
>> 
>> It looks like you're hitting https://github.com/beekhof/pacemaker/commit/58962338
>> Since you appear to be on rhel6 (or a clone of rhel6), can I suggest you use the 1.1.10 packages that come with 6.4?
>> They include the above patch.
> 
> I already use (builded from source two weeks ago)

Upstream 1.1.10 does not include the above patch.

> * pacemaker 1.1.10
> * resource-agents 3.9.5
> * corosync 2.3.2
> * libqb 0.16
> & CentOS 6.4
> 
> The same config work on pacemaker 1.1.9/corosync 1.4.5
> Not ideal, but no such problem.
> 
> At first idea - I thought I should move the target-role=Master from MS to primitive pgsql.
> And so even working.
> But after a crash killing the main PostgreSQL process - started the same.
> Today's experiments showed that this behavior starts after I add in MS "notify=true".
> But primitive pgsql not properly work without "notify" messages.
> While I in frustration :(
> 
> 
>> Also, just to be sure. Are you expecting monitor operations to detect when you started a resource manually?
>> If so, you'll need a monitor operation with role=Stopped. We don't do that by default.
> 
> I expect that the resource monitoring on all the time, otherwise how to control them?

When a node joins the cluster, we check to see if it had any resources running.
If no-one has the resource running, we pick a node and start it there.

If a malicious admin then starts the resource somewhere else, manually, we would not normally detect this.
It is assumed that someone trusted with root privileges would not do this on purpose.

However, if you do not trust your admins, as explained above, you can configure pacemaker to periodically re-check the node to detect and recover from this situation.

> Or I do not quite understand the question.
> 
> 
>>>>  'crm_mon -o' might be a good source of information too.
>>>  Therefore, I see that my resources allegedly functioning normally.
>>> 
>>>  # crm_mon -o1
>>>  Last updated: Tue Nov 12 09:27:16 2013
>>>  Last change: Tue Nov 12 00:08:35 2013 via crm_attribute on dev-cluster2-node2
>>>  Stack: corosync
>>>  Current DC: dev-cluster2-node2 (172793105) - partition with quorum
>>>  Version: 1.1.10-1.el6-368c726
>>>  3 Nodes configured
>>>  337 Resources configured
>>> 
>>>  Online: [ dev-cluster2-node2 dev-cluster2-node3 dev-cluster2-node4 ]
>>> 
>>>  Clone Set: clonePing [pingCheck]
>>>      Started: [ dev-cluster2-node2 dev-cluster2-node3 dev-cluster2-node4 ]
>>>  Master/Slave Set: msPgsql [pgsql]
>>>      Masters: [ dev-cluster2-node2 ]
>>>      Slaves: [ dev-cluster2-node3 dev-cluster2-node4 ]
>>>  VirtualIP      (ocf::heartbeat:IPaddr2):       Started dev-cluster2-node2
>>> 
>>>  Operations:
>>>  * Node dev-cluster2-node2:
>>>    pingCheck: migration-threshold=1
>>>     + (20) start: rc=0 (ok)
>>>     + (23) monitor: interval=10000ms rc=0 (ok)
>>>    pgsql: migration-threshold=1
>>>     + (41) promote: rc=0 (ok)
>>>     + (87) monitor: interval=1000ms rc=8 (master)
>>>    VirtualIP: migration-threshold=1
>>>     + (49) start: rc=0 (ok)
>>>     + (52) monitor: interval=10000ms rc=0 (ok)
>>>  * Node dev-cluster2-node3:
>>>    pingCheck: migration-threshold=1
>>>     + (20) start: rc=0 (ok)
>>>     + (23) monitor: interval=10000ms rc=0 (ok)
>>>    pgsql: migration-threshold=1
>>>     + (26) start: rc=0 (ok)
>>>     + (32) monitor: interval=10000ms rc=0 (ok)
>>>  * Node dev-cluster2-node4:
>>>    pingCheck: migration-threshold=1
>>>     + (20) start: rc=0 (ok)
>>>     + (23) monitor: interval=10000ms rc=0 (ok)
>>>    pgsql: migration-threshold=1
>>>     + (26) start: rc=0 (ok)
>>>     + (32) monitor: interval=10000ms rc=0 (ok)
>>> 
>>>  In reality now killed (signal 4|6) the PG master and the penultimate slave PG.
>>>  IMHO, even if I have something configured incorrectly, the inability to monitor the resource must cause a fatal error.
>>>  Or is there a reason not to do so?
>>> 
>>>  _______________________________________________
>>>  Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>  http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> 
>>>  Project Home: http://www.clusterlabs.org
>>>  Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>  Bugs: http://bugs.clusterlabs.org
>> 
>> ,
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20131114/e80b973b/attachment-0004.sig>

More information about the Pacemaker mailing list