[Pacemaker] pacemakerd does not daemonize?

Wed Apr 10 00:21:55 EDT 2013

On Apr 10, 2013, at 2:06 , Andrew Beekhof <andrew at beekhof.net> wrote:

> 
> On 09/04/2013, at 4:13 PM, Andrei Belov <defanator at gmail.com> wrote:
> 
>> Hello pacemaker users,
>> 
>> I noticed that neither "-p" nor "-f" option does not make any sense for pacemakerd -
>> pid_file is never used, and "f" option marked as "Legacy".
>> 
>> Is the ability to run as a daemon disappeared completely?
> 
> Is "pacemakerd &" insufficient?
> This is what the init script uses.

That's ok, I just was a little confused by meaningless options in "pacemakerd --help".

>> Also I'd like to know if there are any reasons to worry about the following:
> 
> Absolutely... four processes crashed/aborted.
> 
>> 
>> Apr 08 19:54:20 [6025] pacemakerd:     info: pcmk_child_exit:   Child process crmd exited (pid=6031, rc=0)
>> Apr 08 19:54:20 [6025] pacemakerd:     info: pcmk_child_exit:   Child process pengine exited (pid=6030, rc=0)
>> Apr 08 19:54:24 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process attrd terminated with signal 6 (pid=6029, core=128)
>> Apr 08 19:54:29 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process lrmd terminated with signal 6 (pid=6028, core=128)
>> Apr 08 19:54:33 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process stonith-ng terminated with signal 6 (pid=6027, core=128)
>> Apr 08 19:54:38 [6025] pacemakerd:   notice: pcmk_child_exit:   Child process cib terminated with signal 6 (pid=6026, core=128)
>> 
>> Why some helper daemons could be terminated using abort() ?
> 
> Something _really_ bad happened.

I suspect something wrong with pacemaker + libqb and QB_IPC_SOCKET.
Would appreciate any advices - my knowledge of pacemaker/libqb internals
is very limited.

It looks like the reason for abort() is somewhere in qb_ipcs_connection_unref():

Core was generated by `/opt/local/libexec/pacemaker/attrd'.
Program terminated with signal 6, Aborted.
#0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
(gdb) bt
#0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
#2  0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
#3  0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
#4  0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
#5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
#6  0x00000000004044f9 in main ()

Core was generated by `/opt/local/libexec/pacemaker/cib'.
Program terminated with signal 6, Aborted.
#0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
(gdb) bt
#0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfffffd7fff0e4ddd in thr_kill () from /lib/64/libc.so.1
#2  0xfffffd7fff07a971 in raise () from /lib/64/libc.so.1
#3  0xfffffd7fff0500a1 in abort () from /lib/64/libc.so.1
#4  0xfffffd7fff0503f5 in _assert () from /lib/64/libc.so.1
#5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
#6  0x0000000000410438 in cib_shutdown ()
#7  0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x49be80, callback=<optimized out>, userdata=<optimized out>)
    at mainloop.c:203
#8  0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
#9  0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
#10 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
#11 0x0000000000411056 in cib_init ()
#12 0x000000000041163e in main ()

Core was generated by `/opt/local/libexec/pacemaker/lrmd'.
Program terminated with signal 6, Aborted.
#0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
(gdb) bt
#0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
#2  0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
#3  0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
#4  0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
#5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
#6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
#7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
#8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
#9  0xfffffd7fbfc55a3f in mainloop_del_ipc_server (server=<optimized out>) at mainloop.c:517
#10 0x00000000004041cd in lrmd_shutdown ()
#11 0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x48ad40, callback=<optimized out>, userdata=<optimized out>)
    at mainloop.c:203
#12 0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
#13 0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
#14 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
#15 0x00000000004045e3 in main ()

Core was generated by `/opt/local/libexec/pacemaker/stonithd'.
Program terminated with signal 6, Aborted.
#0  0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
(gdb) bt
#0  0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
#1  0xfffffd7fff104ddd in thr_kill () from /lib/64/libc.so.1
#2  0xfffffd7fff09a971 in raise () from /lib/64/libc.so.1
#3  0xfffffd7fff0700a1 in abort () from /lib/64/libc.so.1
#4  0xfffffd7fff0703f5 in _assert () from /lib/64/libc.so.1
#5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
#6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
#7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
#8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
#9  0x0000000000405e60 in ?? ()
#10 0x0000000000407d28 in main ()

>> Here's the backtrace for cib:
>> 
>> Core was generated by `/opt/local/libexec/pacemaker/cib'.
>> Program terminated with signal 6, Aborted.
>> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>> (gdb) bt
>> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>> #1  0xfffffd7fff0e4ddd in thr_kill () from /lib/64/libc.so.1
>> #2  0xfffffd7fff07a971 in raise () from /lib/64/libc.so.1
>> #3  0xfffffd7fff0500a1 in abort () from /lib/64/libc.so.1
>> #4  0xfffffd7fff0503f5 in _assert () from /lib/64/libc.so.1
>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>> #6  0x0000000000410438 in cib_shutdown ()
>> #7  0xfffffd7fbff2533f in crm_signal_dispatch (source=0x49be80, callback=<optimized out>, userdata=<optimized out>)
>>   at mainloop.c:203
>> #8  0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
>> #9  0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
>> #10 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
>> #11 0x0000000000411056 in cib_init ()
>> #12 0x000000000041163e in main ()
>> (gdb) 
>> 
>> 
>> I'm using pacemaker 1.1.8 under SunOS (Joyent cloud).
>> 
>> 
>> Best regards,
>> Andrei