[Pacemaker] node status does not change even if pacemakerd dies

Andrew Beekhof andrew at beekhof.net
Wed Dec 5 00:48:20 UTC 2012 

On Wednesday, December 5, 2012, David Vossel wrote:

>
>
> ----- Original Message -----
> > From: "Kazunori INOUE" <inouekazu at intellilink.co.jp <javascript:;>>
> > To: "The Pacemaker cluster resource manager" <
> pacemaker at oss.clusterlabs.org>
> > Sent: Monday, December 3, 2012 11:41:56 PM
> > Subject: Re: [Pacemaker] node status does not change even if pacemakerd
> dies
> >
> > (12.12.03 20:24), Andrew Beekhof wrote:
> > > On Mon, Dec 3, 2012 at 8:15 PM, Kazunori INOUE
> > > <inouekazu at intellilink.co.jp> wrote:
> > >> (12.11.30 23:52), David Vossel wrote:
> > >>>
> > >>> ----- Original Message -----
> > >>>>
> > >>>> From: "Kazunori INOUE" <inouekazu at intellilink.co.jp>
> > >>>> To: "pacemaker at oss" <pacemaker at oss.clusterlabs.org>
> > >>>> Sent: Friday, November 30, 2012 2:38:50 AM
> > >>>> Subject: [Pacemaker] node status does not change even if
> > >>>> pacemakerd dies
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I am testing the latest version.
> > >>>> - ClusterLabs/pacemaker  9c13d14640(Nov 27, 2012)
> > >>>> - corosync               92e0f9c7bb(Nov 07, 2012)
> > >>>> - libqb                  30a7871646(Nov 29, 2012)
> > >>>>
> > >>>>
> > >>>> Although I killed pacemakerd, node status did not change.
> > >>>>
> > >>>>    [dev1 ~]$ pkill -9 pacemakerd
> > >>>>    [dev1 ~]$ crm_mon
> > >>>>      :
> > >>>>    Stack: corosync
> > >>>>    Current DC: dev2 (2472913088) - partition with quorum
> > >>>>    Version: 1.1.8-9c13d14
> > >>>>    2 Nodes configured, unknown expected votes
> > >>>>    0 Resources configured.
> > >>>>
> > >>>>
> > >>>>    Online: [ dev1 dev2 ]
> > >>>>
> > >>>>    [dev1 ~]$ ps -ef|egrep 'corosync|pacemaker'
> > >>>>    root     11990     1  1 16:05 ?        00:00:00 corosync
> > >>>>    496      12010     1  0 16:05 ?        00:00:00
> > >>>>    /usr/libexec/pacemaker/cib
> > >>>>    root     12011     1  0 16:05 ?        00:00:00
> > >>>>    /usr/libexec/pacemaker/stonithd
> > >>>>    root     12012     1  0 16:05 ?        00:00:00
> > >>>>    /usr/libexec/pacemaker/lrmd
> > >>>>    496      12013     1  0 16:05 ?        00:00:00
> > >>>>    /usr/libexec/pacemaker/attrd
> > >>>>    496      12014     1  0 16:05 ?        00:00:00
> > >>>>    /usr/libexec/pacemaker/pengine
> > >>>>    496      12015     1  0 16:05 ?        00:00:00
> > >>>>    /usr/libexec/pacemaker/crmd
> > >>>>
> > >>>>
> > >>>> We want the node status to change to
> > >>>> OFFLINE(stonith-enabled=false),
> > >>>> UNCLEAN(stonith-enabled=true).
> > >>>> That is, we want the function of this deleted code.
> > >>>>
> > >>>>
> https://github.com/ClusterLabs/pacemaker/commit/dfdfb6c9087e644cb898143e198b240eb9a928b4
> > >>>
> > >>>
> > >>> How are you launching pacemakerd?  The systemd service script
> > >>> relaunches
> > >>> pacemakerd on failure and pacemakerd has the ability to attach to
> > >>> all the
> > >>> old processes if they are still around as if nothiAh yes, that is a
> problem.
>
> Having pacemaker still running when the init script says it is down...
> that is bad.  Perhaps we should just make the init script smart enough to
> check to make sure all the pacemaker components are down after pacemakerd
> is down.
>
> The argument of whether or not the failure of pacemakerd is something that
> the cluster should be alerted to is something i'm not sure about.  With the
> corosync 2.0 stack, pacemakerd really doesn't do anything except launch
> processes/relaunch processes.  A cluster can be completely functional
> without a pacemakerd instance running anywhere.  If any of the actual
> pacemaker components on a node fail, the logic that causes that node to get
> fenced has nothing to do with pacemakerd.


The point about init being confused is valid, but I'd prefer to find a way
to get the right info than to fix the problem by causing more downtime.

Having pacemakerd reattach to the existing services when it gets respawned
works for systemd, but we should probably make the LSB status check smarter
too.


>
> -- Vossel
>
>
> > > This isnt the case when the plugin is in use though, but then I'd
> > > also
> > > have expected most of the processes to die also.
> > >
> > Since node status will also change if such a result is brought,
> > we desire to become so.
> >
> > >>
> > >> ----
> > >> $ cat /etc/redhat-release
> > >> Red Hat Enterprise Linux Server release 6.3 (Santiago)
> > >>
> > >> $ ./configure --sysconfdir=/etc --localstatedir=/var
> > >> --without-cman
> > >> --without-heartbeat
> > >> -snip-
> > >> pacemaker configuration:
> > >>    Version                  = 1.1.8 (Build: 9c13d14)
> > >>    Features                 = generated-manpages agent-manpages
> > >>    ascii-docs
> > >> publican-docs ncurses libqb-logging libqb-ipc lha-fencing
> > >>  corosync-native
> > >> snmp
> > >>
> > >>
> > >> $ cat config.log
> > >> -snip-
> > >> 6000 | #define BUILD_VERSION "9c13d14"
> > >> 6001 | /* end confdefs.h.  */
> > >> 6002 | #include <gio/gio.h>
> > >> 6003 |
> > >> 6004 | int
> > >> 6005 | main ()
> > >> 6006 | {
> > >> 6007 | if (sizeof (GDBusProxy))
> > >> 6008 |        return 0;
> > >> 6009 |   ;
> > >> 6010 |   return 0;
> > >> 6011 | }
> > >> 6012 configure:32411: result: no
> > >> 6013 configure:32417: WARNING: Unable to support systemd/upstart.
> > >> You need
> > >> to use glib >= 2.26
> > >> -snip-
> > >> 6286 | #define BUILD_VERSION "9c13d14"
> > >> 6287 | #define SUPPORT_UPSTART 0
> > >> 6288 | #define SUPPORT_SYSTEMD 0
> > >>
> > >>
> > >> Best Regards,
> > >> Kazunori INOUE
> > >>
> > >>
> > >>>
> > >>>> related bugzilla:
> > >>>> http://bugs.clusterlabs.org/show_bug.cgi?id=5064
> > >>>>
> > >>>>
> > >>>> Best Regards,
> > >>>> Kazunori INOUE
> > >>>>
> > >>>> _______________________________________________
> > >>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >>>>
> > >>>> Project Home: http://www.clusterlabs.org
> > >>>> Getting started:
> > >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >>>> Bugs: http://bugs.clusterlabs.org
> > >>>>
> > >>>
> > >>> _______________________________________________
> > >>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >>>
> > >>> Project Home: http://www.clusterlabs.org
> > >>> Getting started:
> > >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >>> Bugs: http://bugs.clusterlabs.org
> > >>
> > >>
> > >> _______________________________________________
> > >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >>
> > >> Project Hom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20121205/f2205803/attachment.htm>

More information about the Pacemaker mailing list