[Pacemaker] pacemakerd does not daemonize?

Wed Apr 10 00:25:37 EDT 2013

On Apr 10, 2013, at 8:21 , Andrei Belov <defanator at gmail.com> wrote:

> 
> On Apr 10, 2013, at 2:06 , Andrew Beekhof <andrew at beekhof.net> wrote:
> 
>> 
>> On 09/04/2013, at 4:13 PM, Andrei Belov <defanator at gmail.com> wrote:
>> 
>>> Hello pacemaker users,
>>> 
>>> I noticed that neither "-p" nor "-f" option does not make any sense for pacemakerd -
>>> pid_file is never used, and "f" option marked as "Legacy".
>>> 
>>> Is the ability to run as a daemon disappeared completely?
>> 
>> Is "pacemakerd &" insufficient?
>> This is what the init script uses.
> 
> That's ok, I just was a little confused by meaningless options in "pacemakerd --help".
> 
> 
>>> Also I'd like to know if there are any reasons to worry about the following:
>> 
>> Absolutely... four processes crashed/aborted.
>> 
>>> 
>>> Apr 08 19:54:20 [6025] pacemakerd:     info: pcmk_child_exit:   Child process crmd exited (pid=6031, rc=0)
>>> Apr 08 19:54:20 [6025] pacemakerd:     info: pcmk_child_exit:   Child process pengine exited (pid=6030, rc=0)
>>> Apr 08 19:54:24 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process attrd terminated with signal 6 (pid=6029, core=128)
>>> Apr 08 19:54:29 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process lrmd terminated with signal 6 (pid=6028, core=128)
>>> Apr 08 19:54:33 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process stonith-ng terminated with signal 6 (pid=6027, core=128)
>>> Apr 08 19:54:38 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process cib terminated with signal 6 (pid=6026, core=128)
>>> 
>>> Why some helper daemons could be terminated using abort() ?
>> 
>> Something _really_ bad happened.
> 
> I suspect something wrong with pacemaker + libqb and QB_IPC_SOCKET.
> Would appreciate any advices - my knowledge of pacemaker/libqb internals
> is very limited.
> 
> It looks like the reason for abort() is somewhere in qb_ipcs_connection_unref():
> 
> Core was generated by `/opt/local/libexec/pacemaker/attrd'.
> Program terminated with signal 6, Aborted.
> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
> (gdb) bt
> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
> #1  0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
> #2  0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
> #3  0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
> #4  0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
> #6  0x00000000004044f9 in main ()
> 
> Core was generated by `/opt/local/libexec/pacemaker/cib'.
> Program terminated with signal 6, Aborted.
> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
> (gdb) bt
> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
> #1  0xfffffd7fff0e4ddd in thr_kill () from /lib/64/libc.so.1
> #2  0xfffffd7fff07a971 in raise () from /lib/64/libc.so.1
> #3  0xfffffd7fff0500a1 in abort () from /lib/64/libc.so.1
> #4  0xfffffd7fff0503f5 in _assert () from /lib/64/libc.so.1
> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
> #6  0x0000000000410438 in cib_shutdown ()
> #7  0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x49be80, callback=<optimized out>, userdata=<optimized out>)
>    at mainloop.c:203
> #8  0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
> #9  0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
> #10 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
> #11 0x0000000000411056 in cib_init ()
> #12 0x000000000041163e in main ()
> 
> Core was generated by `/opt/local/libexec/pacemaker/lrmd'.
> Program terminated with signal 6, Aborted.
> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
> (gdb) bt
> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
> #1  0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
> #2  0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
> #3  0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
> #4  0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
> #6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
> #7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
> #8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
> #9  0xfffffd7fbfc55a3f in mainloop_del_ipc_server (server=<optimized out>) at mainloop.c:517
> #10 0x00000000004041cd in lrmd_shutdown ()
> #11 0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x48ad40, callback=<optimized out>, userdata=<optimized out>)
>    at mainloop.c:203
> #12 0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
> #13 0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
> #14 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
> #15 0x00000000004045e3 in main ()
> 
> Core was generated by `/opt/local/libexec/pacemaker/stonithd'.
> Program terminated with signal 6, Aborted.
> #0  0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
> (gdb) bt
> #0  0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
> #1  0xfffffd7fff104ddd in thr_kill () from /lib/64/libc.so.1
> #2  0xfffffd7fff09a971 in raise () from /lib/64/libc.so.1
> #3  0xfffffd7fff0700a1 in abort () from /lib/64/libc.so.1
> #4  0xfffffd7fff0703f5 in _assert () from /lib/64/libc.so.1
> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
> #6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
> #7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
> #8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
> #9  0x0000000000405e60 in ?? ()
> #10 0x0000000000407d28 in main ()

Corresponding error messages from libqb are:

Apr 10 04:05:06 [47131]      attrd:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47131-47133-11)
Apr 10 04:05:13 [47130]       lrmd:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47130-47133-9)
Apr 10 04:05:20 [47129] stonith-ng:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47129-47133-13)
Apr 10 04:05:27 [47128]        cib:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47128-47131-19)

>>> Here's the backtrace for cib:
>>> 
>>> Core was generated by `/opt/local/libexec/pacemaker/cib'.
>>> Program terminated with signal 6, Aborted.
>>> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>>> (gdb) bt
>>> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>>> #1  0xfffffd7fff0e4ddd in thr_kill () from /lib/64/libc.so.1
>>> #2  0xfffffd7fff07a971 in raise () from /lib/64/libc.so.1
>>> #3  0xfffffd7fff0500a1 in abort () from /lib/64/libc.so.1
>>> #4  0xfffffd7fff0503f5 in _assert () from /lib/64/libc.so.1
>>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>>> #6  0x0000000000410438 in cib_shutdown ()
>>> #7  0xfffffd7fbff2533f in crm_signal_dispatch (source=0x49be80, callback=<optimized out>, userdata=<optimized out>)
>>>  at mainloop.c:203
>>> #8  0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
>>> #9  0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
>>> #10 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
>>> #11 0x0000000000411056 in cib_init ()
>>> #12 0x000000000041163e in main ()
>>> (gdb) 
>>> 
>>> 
>>> I'm using pacemaker 1.1.8 under SunOS (Joyent cloud).
>>> 
>>> 
>>> Best regards,
>>> Andrei
> 