[Pacemaker] g_hash_table_lookup: assertion `hash_table != NULL' failed

[Andrew Beekhof]

On Mon, Sep 13, 2010 at 5:40 PM, Vladislav Bogdanov
<bubble at hoster-ok.com> wrote:
> Hi Andrew, hi all,
>
> Upgraded to todays tip and see bunch of assertion messages in logs
> (together with some segfaults from pengine).

tip?  From the 1.1 repo?

>
> Here is what I see:
>
> Sep 13 16:53:43 s01-1 pengine: [4120]: ERROR: crm_abort: xpath_search:
> Triggered assert at xml.c:2599 : xml_top != NULL
> Sep 13 16:56:14 s01-1 pengine: [7075]: ERROR: crm_abort: xpath_search:
> Triggered assert at xml.c:2599 : xml_top != NULL
> Sep 13 17:00:20 s01-1 pengine: [9281]: ERROR: crm_abort: xpath_search:
> Triggered assert at xml.c:2599 : xml_top != NULL
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14536 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14537 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14538 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14540 to record non-fatal assert at utils.c:452 :
> g_hash_table_destroy: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14546 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14565 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14581 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:23 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14584 to record non-fatal assert at utils.c:452 :
> g_hash_table_destroy: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14731 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14732 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14733 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14735 to record non-fatal assert at utils.c:452 :
> g_hash_table_destroy: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14771 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14772 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14773 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:01:24 s01-1 crmd: [14368]: ERROR: crm_abort: crm_glib_handler:
> Forked child 14775 to record non-fatal assert at utils.c:452 :
> g_hash_table_destroy: assertion `hash_table != NULL' failed
> Sep 13 17:22:19 s01-1 pengine: [14367]: ERROR: crm_abort: xpath_search:
> Triggered assert at xml.c:2599 : xml_top != NULL
> Sep 13 17:23:22 s01-1 crmd: [1133]: ERROR: crm_abort: crm_glib_handler:
> Forked child 1532 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:23:22 s01-1 crmd: [1133]: ERROR: crm_abort: crm_glib_handler:
> Forked child 1545 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:23:22 s01-1 crmd: [1133]: ERROR: crm_abort: crm_glib_handler:
> Forked child 1560 to record non-fatal assert at utils.c:452 :
> g_hash_table_lookup: assertion `hash_table != NULL' failed
> Sep 13 17:23:22 s01-1 crmd: [1133]: ERROR: crm_abort: crm_glib_handler:
> Forked child 1565 to record non-fatal assert at utils.c:452 :
> g_hash_table_destroy: assertion `hash_table != NULL' failed
>
> and
> pengine[14367]: segfault at 8 ip 00007f6f310abf89 sp 00007fff5879f1d0
> error 4 in libpengine.so.3.0.0[7f6f31099000+5a000]
> pengine[1132]: segfault at 8 ip 00007fef1f6a7f89 sp 00007fff0ab135e0
> error 4 in libpengine.so.3.0.0[7fef1f695000+5a000]
>
> Ah, these are connected to xml_top assert.
>
> This happens for both nodes and didn't happen before, so perhaps it
> connected to latest commits.
>
> Should I provide some more info?

Yes, a hb_report please.
It should produce some stack traces and input files for use to
diagnose the issue.