[Pacemaker] node status does not change even if pacemakerd dies

Andrew Beekhof andrew at beekhof.net
Wed Dec 12 23:26:31 UTC 2012 

On Wed, Dec 12, 2012 at 8:02 PM, Kazunori INOUE
<inouekazu at intellilink.co.jp> wrote:
>
> Hi,
>
> I recognize that pacemakerd is much less likely to crash.
> However, a possibility of being killed by OOM_Killer etc. is not 0%.

True.  Although we just established in another thread that we don't
have any leaks :)

> So I think that a user gets confused. since behavior at the time of process
> death differs even if pacemakerd is running.
>
> case A)
>  When pacemakerd and other processes (crmd etc.) are the parent-child
> relation.
>

[snip]

>
>  For example, crmd died.
>  However, since it is relaunched, the state of the cluster is not affected.

Right.

[snip]

>
> case B)
>  When pacemakerd and other processes are NOT the parent-child relation.
>  Although pacemakerd was killed, it assumed the state where it was respawned
> by Upstart.
>
>   $ service corosync start ; service pacemaker start
>   $ pkill -9 pacemakerd
>   $ ps -ef|egrep 'corosync|pacemaker|UID'
>   UID      PID  PPID  C STIME TTY       TIME CMD
>   root   21091     1  1 14:52 ?     00:00:00 corosync
>   496    21099     1  0 14:52 ?     00:00:00 /usr/libexec/pacemaker/cib
>   root   21100     1  0 14:52 ?     00:00:00 /usr/libexec/pacemaker/stonithd
>   root   21101     1  0 14:52 ?     00:00:00 /usr/libexec/pacemaker/lrmd
>   496    21102     1  0 14:52 ?     00:00:00 /usr/libexec/pacemaker/attrd
>   496    21103     1  0 14:52 ?     00:00:00 /usr/libexec/pacemaker/pengine
>   496    21104     1  0 14:52 ?     00:00:00 /usr/libexec/pacemaker/crmd
>   root   21128     1  1 14:53 ?     00:00:00 /usr/sbin/pacemakerd

Yep, looks right.

>  In this case, the node will be set to UNCLEAN if crmd dies.
>  That is, the node will be fenced if there is stonith resource.

Which is exactly what happens if only pacemakerd is killed with your proposal.
Except now you have time to do a graceful pacemaker restart to
re-establish the parent-child relationship.

If you want to compare B with something, it needs to be with the old
"children terminate if pacemakerd dies" strategy.
Which is:

>   $ service corosync start ; service pacemaker start
>   $ pkill -9 pacemakerd
>  ... the node will be set to UNCLEAN

Old way: always downtime because children terminate which triggers fencing
Our way: no downtime unless there is an additional failure (to the cib or crmd)

Given that we're trying for HA, the second seems preferable.

>
>   $ pkill -9 crmd
>   $ crm_mon -1
>   Last updated: Wed Dec 12 14:53:48 2012
>   Last change: Wed Dec 12 14:53:10 2012 via crmd on dev2
>
>   Stack: corosync
>   Current DC: dev2 (2472913088) - partition with quorum
>   Version: 1.1.8-3035414
>
>   2 Nodes configured, unknown expected votes
>   0 Resources configured.
>
>   Node dev1 (2506467520): UNCLEAN (online)
>   Online: [ dev2 ]
>
>
> How about making behavior selectable with an option?

MORE_DOWNTIME_PLEASE=(true|false) ?

>
> When pacemakerd dies,
> mode A) which behaves in an existing way. (default)
> mode B) which makes the node UNCLEAN.
>
> Best Regards,
> Kazunori INOUE
>
>
>
>> Making stop work when there is no pacemakerd process is a different
>> matter. We can make that work.
>>
>>>
>>> Though the best solution is to relaunch pacemakerd, if it is difficult,
>>> I think that a shortcut method is to make a node unclean.
>>>
>>>
>>> And now, I tried Upstart a little bit.
>>>
>>> 1) started the corosync and pacemaker.
>>>
>>>   $ cat /etc/init/pacemaker.conf
>>>   respawn
>>>   script
>>>       [ -f /etc/sysconfig/pacemaker ] && {
>>>           . /etc/sysconfig/pacemaker
>>>       }
>>>       exec /usr/sbin/pacemakerd
>>>   end script
>>>
>>>   $ service co start
>>>   Starting Corosync Cluster Engine (corosync):               [  OK  ]
>>>   $ initctl start pacemaker
>>>   pacemaker start/running, process 4702
>>>
>>>
>>>   $ ps -ef|egrep 'corosync|pacemaker'
>>>   root   4695     1  0 17:21 ?    00:00:00 corosync
>>>   root   4702     1  0 17:21 ?    00:00:00 /usr/sbin/pacemakerd
>>>   496    4703  4702  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/cib
>>>   root   4704  4702  0 17:21 ?    00:00:00
>>> /usr/libexec/pacemaker/stonithd
>>>   root   4705  4702  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/lrmd
>>>   496    4706  4702  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/attrd
>>>   496    4707  4702  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/pengine
>>>   496    4708  4702  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/crmd
>>>
>>> 2) killed pacemakerd.
>>>
>>>   $ pkill -9 pacemakerd
>>>
>>>   $ ps -ef|egrep 'corosync|pacemaker'
>>>   root   4695     1  0 17:21 ?    00:00:01 corosync
>>>   496    4703     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/cib
>>>   root   4704     1  0 17:21 ?    00:00:00
>>> /usr/libexec/pacemaker/stonithd
>>>   root   4705     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/lrmd
>>>   496    4706     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/attrd
>>>   496    4707     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/pengine
>>>   496    4708     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/crmd
>>>   root   4760     1  1 17:24 ?    00:00:00 /usr/sbin/pacemakerd
>>>
>>> 3) then I stopped pacemakerd. however, some processes did not stop.
>>>
>>>   $ initctl stop pacemaker
>>>   pacemaker stop/waiting
>>>
>>>
>>>   $ ps -ef|egrep 'corosync|pacemaker'
>>>   root   4695     1  0 17:21 ?    00:00:01 corosync
>>>   496    4703     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/cib
>>>   root   4704     1  0 17:21 ?    00:00:00
>>> /usr/libexec/pacemaker/stonithd
>>>   root   4705     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/lrmd
>>>   496    4706     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/attrd
>>>   496    4707     1  0 17:21 ?    00:00:00 /usr/libexec/pacemaker/pengine
>>>
>>> Best Regards,
>>> Kazunori INOUE
>>>
>>>
>>>>>> This isnt the case when the plugin is in use though, but then I'd
>>>>>> also
>>>>>> have expected most of the processes to die also.
>>>>>>
>>>>> Since node status will also change if such a result is brought,
>>>>> we desire to become so.
>>>>>
>>>>>>>
>>>>>>> ----
>>>>>>> $ cat /etc/redhat-release
>>>>>>> Red Hat Enterprise Linux Server release 6.3 (Santiago)
>>>>>>>
>>>>>>> $ ./configure --sysconfdir=/etc --localstatedir=/var
>>>>>>> --without-cman
>>>>>>> --without-heartbeat
>>>>>>> -snip-
>>>>>>> pacemaker configuration:
>>>>>>>      Version                  = 1.1.8 (Build: 9c13d14)
>>>>>>>      Features                 = generated-manpages agent-manpages
>>>>>>>      ascii-docs
>>>>>>> publican-docs ncurses libqb-logging libqb-ipc lha-fencing
>>>>>>>    corosync-native
>>>>>>> snmp
>>>>>>>
>>>>>>>
>>>>>>> $ cat config.log
>>>>>>> -snip-
>>>>>>> 6000 | #define BUILD_VERSION "9c13d14"
>>>>>>> 6001 | /* end confdefs.h.  */
>>>>>>> 6002 | #include <gio/gio.h>
>>>>>>> 6003 |
>>>>>>> 6004 | int
>>>>>>> 6005 | main ()
>>>>>>> 6006 | {
>>>>>>> 6007 | if (sizeof (GDBusProxy))
>>>>>>> 6008 |        return 0;
>>>>>>> 6009 |   ;
>>>>>>> 6010 |   return 0;
>>>>>>> 6011 | }
>>>>>>> 6012 configure:32411: result: no
>>>>>>> 6013 configure:32417: WARNING: Unable to support systemd/upstart.
>>>>>>> You need
>>>>>>> to use glib >= 2.26
>>>>>>> -snip-
>>>>>>> 6286 | #define BUILD_VERSION "9c13d14"
>>>>>>> 6287 | #define SUPPORT_UPSTART 0
>>>>>>> 6288 | #define SUPPORT_SYSTEMD 0
>>>>>>>
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Kazunori INOUE
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> related bugzilla:
>>>>>>>>> http://bugs.clusterlabs.org/show_bug.cgi?id=5064
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Kazunori INOUE
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>
>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>> Getting started:
>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>
>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>> Getting started:
>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>
>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>> Getting started:
>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list