[Pacemaker] [Problem]When Pacemaker uses a new version of glib, g_source_remove fails.
renayama19661014 at ybb.ne.jp
renayama19661014 at ybb.ne.jp
Fri Oct 10 05:45:26 UTC 2014
Hi Andrew,
I applied three corrections that you made and checked movement.
I picked all "abort" processing with g_source_remove() of services.c just to make sure.
* I set following "abort" in four places that carried out g_source_remove
>>> if (g_source_remove(op->opaque->repeat_timer) == FALSE)
> {
>>> abort();
>>> }
As a result, "abort" still occurred.
The problem does not seem to be yet settled by your correction.
(gdb) where
#0 0x00007fdd923e1f79 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fdd923e5388 in __GI_abort () at abort.c:89
#2 0x00007fdd92b9fe77 in crm_abort (file=file at entry=0x7fdd92bd352b "logging.c",
function=function at entry=0x7fdd92bd48c0 <__FUNCTION__.23262> "crm_glib_handler", line=line at entry=73,
assert_condition=assert_condition at entry=0xe20b80 "Source ID 40 was not found when attempting to remove it", do_core=do_core at entry=1,
do_fork=<optimized out>, do_fork at entry=1) at utils.c:1195
#3 0x00007fdd92bc7ca7 in crm_glib_handler (log_domain=0x7fdd92130b6e "GLib", flags=<optimized out>,
message=0xe20b80 "Source ID 40 was not found when attempting to remove it", user_data=<optimized out>) at logging.c:73
#4 0x00007fdd920f2ae1 in g_logv () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5 0x00007fdd920f2d72 in g_log () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6 0x00007fdd920eac5c in g_source_remove () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#7 0x00007fdd92984b55 in cancel_recurring_action (op=op at entry=0xe19b90) at services.c:365
#8 0x00007fdd92984bee in services_action_cancel (name=name at entry=0xe1d2d0 "dummy2", action=<optimized out>, interval=interval at entry=10000)
at services.c:387
#9 0x000000000040405a in cancel_op (rsc_id=rsc_id at entry=0xe1d2d0 "dummy2", action=action at entry=0xe10d90 "monitor", interval=10000)
at lrmd.c:1404
#10 0x000000000040614f in process_lrmd_rsc_cancel (client=0xe17290, id=74, request=0xe1be10) at lrmd.c:1468
#11 process_lrmd_message (client=client at entry=0xe17290, id=74, request=request at entry=0xe1be10) at lrmd.c:1507
#12 0x0000000000402bac in lrmd_ipc_dispatch (c=0xe169c0, data=<optimized out>, size=361) at main.c:148
#13 0x00007fdd91e4d4d9 in qb_ipcs_dispatch_connection_request () from /usr/lib/libqb.so.0
#14 0x00007fdd92bc409d in gio_read_socket (gio=<optimized out>, condition=G_IO_IN, data=0xe158a8) at mainloop.c:437
#15 0x00007fdd920ebce5 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
---Type <return> to continue, or q <return> to quit---
#16 0x00007fdd920ec048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007fdd920ec30a in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x0000000000402774 in main (argc=<optimized out>, argv=0x7fff22cac268) at main.c:344
Best Regards,
Hideo Yamauchi.
----- Original Message -----
> From: "renayama19661014 at ybb.ne.jp" <renayama19661014 at ybb.ne.jp>
> To: Andrew Beekhof <andrew at beekhof.net>; The Pacemaker cluster resource manager <pacemaker at oss.clusterlabs.org>
> Cc:
> Date: 2014/10/10, Fri 10:55
> Subject: Re: [Pacemaker] [Problem]When Pacemaker uses a new version of glib, g_source_remove fails.
>
> Hi Andrew,
>
> Okay!
>
> I test your patch.
> And I inform you of a result.
>
> Many thanks!
> Hideo Yamauchi.
>
>
>
> ----- Original Message -----
>> From: Andrew Beekhof <andrew at beekhof.net>
>> To: renayama19661014 at ybb.ne.jp; The Pacemaker cluster resource manager
> <pacemaker at oss.clusterlabs.org>
>> Cc:
>> Date: 2014/10/10, Fri 10:47
>> Subject: Re: [Pacemaker] [Problem]When Pacemaker uses a new version of
> glib, g_source_remove fails.
>>
>> Perfect!
>>
>> Can you try this:
>>
>> diff --git a/lib/services/services.c b/lib/services/services.c
>> index 8590b56..cb0f0ae 100644
>> --- a/lib/services/services.c
>> +++ b/lib/services/services.c
>> @@ -417,6 +417,7 @@ services_action_kick(const char *name, const char
> *action,
>> int interval /* ms */
>> free(id);
>>
>> if (op == NULL) {
>> + op->opaque->repeat_timer = 0;
>> return FALSE;
>> }
>>
>> @@ -425,6 +426,7 @@ services_action_kick(const char *name, const char
> *action,
>> int interval /* ms */
>> } else {
>> if (op->opaque->repeat_timer) {
>> g_source_remove(op->opaque->repeat_timer);
>> + op->opaque->repeat_timer = 0;
>> }
>> recurring_action_timer(op);
>> return TRUE;
>> @@ -459,6 +461,7 @@ handle_duplicate_recurring(svc_action_t * op, void
>> (*action_callback) (svc_actio
>> if (dup->pid != 0) {
>> if (op->opaque->repeat_timer) {
>> g_source_remove(op->opaque->repeat_timer);
>> + op->opaque->repeat_timer = 0;
>> }
>> recurring_action_timer(dup);
>> }
>>
>>
>> On 10 Oct 2014, at 12:16 pm, renayama19661014 at ybb.ne.jp wrote:
>>
>>> Hi Andrew,
>>>
>>> Setting of gdb of the Ubuntu environment does not yet go well and I
> touch
>> lrmd and cannot acquire trace.
>>> Please wait for this a little more.
>>>
>>>
>>> But.. I let lrmd terminate abnormally when g_source_remove() of
>> cancel_recurring_action() returned FALSE.
>>> -----
>>> gboolean
>>> cancel_recurring_action(svc_action_t * op)
>>> {
>>> crm_info("Cancelling operation %s", op->id);
>>>
>>> if (recurring_actions) {
>>> g_hash_table_remove(recurring_actions, op->id);
>>> }
>>>
>>> if (op->opaque->repeat_timer) {
>>> if (g_source_remove(op->opaque->repeat_timer) == FALSE)
> {
>>> abort();
>>> }
>>> (snip)
>>> -------core----
>>> #0 0x00007f30aa60ff79 in __GI_raise (sig=sig at entry=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>>>
>>> 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or
> directory.
>>> (gdb) where
>>> #0 0x00007f30aa60ff79 in __GI_raise (sig=sig at entry=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>>> #1 0x00007f30aa613388 in __GI_abort () at abort.c:89
>>> #2 0x00007f30aadcde77 in crm_abort (file=file at entry=0x7f30aae0152b
>> "logging.c",
>>> function=function at entry=0x7f30aae028c0 <__FUNCTION__.23262>
>> "crm_glib_handler", line=line at entry=73,
>>> assert_condition=assert_condition at entry=0x19d2ad0 "Source ID
> 63
>> was not found when attempting to remove it", do_core=do_core at entry=1,
>>> do_fork=<optimized out>, do_fork at entry=1) at utils.c:1195
>>> #3 0x00007f30aadf5ca7 in crm_glib_handler (log_domain=0x7f30aa35eb6e
>> "GLib", flags=<optimized out>,
>>> message=0x19d2ad0 "Source ID 63 was not found when attempting
> to
>> remove it", user_data=<optimized out>) at logging.c:73
>>> #4 0x00007f30aa320ae1 in g_logv () from
>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>> #5 0x00007f30aa320d72 in g_log () from
>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>> #6 0x00007f30aa318c5c in g_source_remove () from
>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>> #7 0x00007f30aabb2b55 in cancel_recurring_action
> (op=op at entry=0x19caa90)
>> at services.c:363
>>> #8 0x00007f30aabb2bee in services_action_cancel
> (name=name at entry=0x19d0530
>> "dummy3", action=<optimized out>,
> interval=interval at entry=10000)
>>> at services.c:385
>>> #9 0x000000000040405a in cancel_op (rsc_id=rsc_id at entry=0x19d0530
>> "dummy3", action=action at entry=0x19cec10 "monitor",
>> interval=10000)
>>> at lrmd.c:1404
>>> #10 0x000000000040614f in process_lrmd_rsc_cancel (client=0x19c8290,
> id=74,
>> request=0x19ca8a0) at lrmd.c:1468
>>> #11 process_lrmd_message (client=client at entry=0x19c8290, id=74,
>> request=request at entry=0x19ca8a0) at lrmd.c:1507
>>> #12 0x0000000000402bac in lrmd_ipc_dispatch (c=0x19c79c0,
>> data=<optimized out>, size=361) at main.c:148
>>> #13 0x00007f30aa07b4d9 in qb_ipcs_dispatch_connection_request () from
>> /usr/lib/libqb.so.0
>>> #14 0x00007f30aadf209d in gio_read_socket (gio=<optimized out>,
>> condition=G_IO_IN, data=0x19c68a8) at mainloop.c:437
>>> #15 0x00007f30aa319ce5 in g_main_context_dispatch () from
>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>> ---Type <return> to continue, or q <return> to quit---
>>> #16 0x00007f30aa31a048 in ?? () from
> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>> #17 0x00007f30aa31a30a in g_main_loop_run () from
>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>> #18 0x0000000000402774 in main (argc=<optimized out>,
>> argv=0x7fffcdd90b88) at main.c:344
>>> ---------
>>>
>>> Best Regards,
>>> Hideo Yamauchi.
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "renayama19661014 at ybb.ne.jp"
>> <renayama19661014 at ybb.ne.jp>
>>>> To: Andrew Beekhof <andrew at beekhof.net>
>>>> Cc: The Pacemaker cluster resource manager
>> <pacemaker at oss.clusterlabs.org>
>>>> Date: 2014/10/7, Tue 11:15
>>>> Subject: Re: [Pacemaker] [Problem]When Pacemaker uses a new
> version of
>> glib, g_source_remove fails.
>>>>
>>>> Hi Andrew,
>>>>
>>>>> Not quite. Returning FALSE from the callback also removes the
>> source from
>>>> glib.
>>>>> So your test case effectively removes t1 twice: once
> implicitly by
>>>> returning
>>>>> FALSE in timer_func1() and then again explicitly in
> timer_func3()
>>>>
>>>>
>>>> Your opinion is right.
>>>>
>>>>
>>>> If Pacemaker repeats and does not remove the resources which timer
>
>> concluded in
>>>> FALSE, glib does not return the error.
>>>>
>>>>
>>>> Many Thanks,
>>>> Hideo Yamauchi.
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: Andrew Beekhof <andrew at beekhof.net>
>>>>> To: renayama19661014 at ybb.ne.jp
>>>>> Cc: The Pacemaker cluster resource manager
>>>> <pacemaker at oss.clusterlabs.org>
>>>>> Date: 2014/10/7, Tue 11:06
>>>>> Subject: Re: [Pacemaker] [Problem]When Pacemaker uses a new
> version
>> of
>>>> glib, g_source_remove fails.
>>>>>
>>>>>
>>>>> On 7 Oct 2014, at 1:03 pm, renayama19661014 at ybb.ne.jp wrote:
>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>>>> These problems seem to be due to a correction of
> next
>> glib
>>>> somehow
>>>>> or
>>>>>>> other.
>>>>>>>> *
>>>>>>>
>>>>>
>>>>
>>
> https://github.com/GNOME/glib/commit/393503ba5bdc7c09cd46b716aaf3d2c63a6c7f9c
>>>>>>>
>>>>>>> The glib behaviour on unbuntu seems reasonable,
> removing
>> a source
>>>>> multiple times
>>>>>>> IS a valid error.
>>>>>>> I need the stack trace to know where/how this
> situation
>> can occur
>>>> in
>>>>> pacemaker.
>>>>>>
>>>>>>
>>>>>> Pacemaker does not remove resources several times as far
> as I
>>
>>>> confirmed it.
>>>>>> In Ubuntu(glib2.40), an error occurs just to remove
> resources
>> first.
>>>>>
>>>>> Not quite. Returning FALSE from the callback also removes the
>> source from
>>>> glib.
>>>>> So your test case effectively removes t1 twice: once
> implicitly by
>>>> returning
>>>>> FALSE in timer_func1() and then again explicitly in
> timer_func3()
>>>>>
>>>>>>
>>>>>> Confirmation and the deletion of resources seem to be
>> necessary not to
>>>>
>>>>> produce an error in Ubuntu.
>>>>>> And this works well in glib of RHEL6.x.(and RHEL7.0)
>>>>>>
>>>>>> if (g_main_context_find_source_by_id (NULL, t1)
> !=
>> NULL) {
>>>>>> g_source_remove(t1);
>>>>>> }
>>>>>>
>>>>>> I send it to you after acquiring stack trace.
>>>>>>
>>>>>> Many Thanks!
>>>>>> Hideo Yamauchi.
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: Andrew Beekhof <andrew at beekhof.net>
>>>>>>> To: renayama19661014 at ybb.ne.jp; The Pacemaker
> cluster
>> resource
>>>> manager
>>>>> <pacemaker at oss.clusterlabs.org>
>>>>>>> Cc:
>>>>>>> Date: 2014/10/7, Tue 09:44
>>>>>>> Subject: Re: [Pacemaker] [Problem]When Pacemaker
> uses a
>> new
>>>> version of
>>>>> glib, g_source_remove fails.
>>>>>>>
>>>>>>>
>>>>>>> On 6 Oct 2014, at 4:09 pm,
> renayama19661014 at ybb.ne.jp
>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> When I move the next sample in
>> RHEL6.5(glib2-2.22.5-7.el6) and
>>>>
>>>>>>> Ubuntu14.04(libglib2.0-0:amd64 2.40.0-2), movement
> is
>> different.
>>>>>>>>
>>>>>>>> * Sample : test2.c
>>>>>>>> {{{
>>>>>>>> #include <stdio.h>
>>>>>>>> #include <stdlib.h>
>>>>>>>> #include <glib.h>
>>>>>>>> #include <sys/times.h>
>>>>>>>> guint t1, t2, t3;
>>>>>>>> gboolean timer_func2(gpointer data){
>>>>>>>> printf("TIMER
> EXPIRE!2\n");
>>>>>>>> fflush(stdout);
>>>>>>>> return FALSE;
>>>>>>>> }
>>>>>>>> gboolean timer_func1(gpointer data){
>>>>>>>> clock_t ret;
>>>>>>>> struct tms buff;
>>>>>>>>
>>>>>>>> ret = times(&buff);
>>>>>>>> printf("TIMER EXPIRE!1
> %d\n",
>>>> (int)ret);
>>>>>>>> fflush(stdout);
>>>>>>>> return FALSE;
>>>>>>>> }
>>>>>>>> gboolean timer_func3(gpointer data){
>>>>>>>> printf("TIMER EXPIRE
> 3!\n");
>>>>>>>> fflush(stdout);
>>>>>>>> printf("remove
> timer1!\n");
>>>>>>>>
>>>>>>>> fflush(stdout);
>>>>>>>> g_source_remove(t1);
>>>>>>>> printf("remove
> timer2!\n");
>>>>>>>> fflush(stdout);
>>>>>>>> g_source_remove(t2);
>>>>>>>> printf("remove
> timer3!\n");
>>>>>>>> fflush(stdout);
>>>>>>>> g_source_remove(t3);
>>>>>>>> return FALSE;
>>>>>>>> }
>>>>>>>> int main(int argc, char** argv){
>>>>>>>> GMainLoop *m;
>>>>>>>> clock_t ret;
>>>>>>>> struct tms buff;
>>>>>>>> gint64 t;
>>>>>>>> m = g_main_new(FALSE);
>>>>>>>> t1 = g_timeout_add(1000, timer_func1,
> NULL);
>>>>>>>> t2 = g_timeout_add(60000, timer_func2,
> NULL);
>>>>>>>> t3 = g_timeout_add(5000, timer_func3,
> NULL);
>>>>>>>> ret = times(&buff);
>>>>>>>> printf("START! %d\n",
>> (int)ret);
>>>>>>>> g_main_run(m);
>>>>>>>> }
>>>>>>>>
>>>>>>>> }}}
>>>>>>>> * Result
>>>>>>>> ---- RHEL6.5(glib2-2.22.5-7.el6) ----
>>>>>>>> [root at snmp1 ~]# ./test2
>>>>>>>> START! 429576012
>>>>>>>> TIMER EXPIRE!1 429576112
>>>>>>>> TIMER EXPIRE 3!
>>>>>>>> remove timer1!
>>>>>>>> remove timer2!
>>>>>>>> remove timer3!
>>>>>>>>
>>>>>>>> ---- Ubuntu14.04(libglib2.0-0:amd64 2.40.0-2)
> ----
>>>>>>>> root at a1be102:~# ./test2
>>>>>>>> START! 1718163089
>>>>>>>> TIMER EXPIRE!1 1718163189
>>>>>>>> TIMER EXPIRE 3!
>>>>>>>> remove timer1!
>>>>>>>>
>>>>>>>> (process:1410): GLib-CRITICAL **: Source ID 1
> was not
>> found
>>>> when
>>>>> attempting
>>>>>>> to remove it
>>>>>>>> remove timer2!
>>>>>>>> remove timer3!
>>>>>>>>
>>>>>>>>
>>>>>>>> These problems seem to be due to a correction of
> next
>> glib
>>>> somehow
>>>>> or
>>>>>>> other.
>>>>>>>> *
>>>>>>>
>>>>>
>>>>
>>
> https://github.com/GNOME/glib/commit/393503ba5bdc7c09cd46b716aaf3d2c63a6c7f9c
>>>>>>>
>>>>>>> The glib behaviour on unbuntu seems reasonable,
> removing
>> a source
>>>>> multiple times
>>>>>>> IS a valid error.
>>>>>>> I need the stack trace to know where/how this
> situation
>> can occur
>>>> in
>>>>> pacemaker.
>>>>>>>
>>>>>>>>
>>>>>>>> In g_source_remove() until before change, the
>> deletion of the
>>>> timer
>>>>> which
>>>>>>> practice completed is possible, but
> g_source_remove()
>> after the
>>>> change
>>>>> causes an
>>>>>>> error.
>>>>>>>>
>>>>>>>> Under this influence, we get the following crit
> error
>> in the
>>>>> environment of
>>>>>>> Pacemaker using a new version of glib.
>>>>>>>>
>>>>>>>> lrmd[1632]: error: crm_abort:
> crm_glib_handler:
>> Forked
>>>> child
>>>>> 1840 to
>>>>>>>> record non-fatal assert at logging.c:73 : Source
> ID
>> 51 was not
>>>>
>>>>> found when
>>>>>>>> attempting to remove it
>>>>>>>> lrmd[1632]: crit: crm_glib_handler: GLib:
> Source
>> ID 51 was
>>>> not
>>>>> found
>>>>>>>> when attempting to remove it
>>>>>>>>
>>>>>>>> It seems that some kind of coping is necessary
> in
>> Pacemaker
>>>> when I
>>>>> think
>>>>>>> about next.
>>>>>>>> * Distribution using a new version of glib
> including
>> Ubuntu.
>>>>>>>> * Version up of future glib of RHEL.
>>>>>>>>
>>>>>>>> A similar problem is reported in the ML.
>>>>>>>> *
>>>>>
> http://www.gossamer-threads.com/lists/linuxha/pacemaker/91333#91333
>>>>>>>> *
>>>> http://www.gossamer-threads.com/lists/linuxha/pacemaker/92408
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Hideo Yamauchi.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pacemaker mailing list:
> Pacemaker at oss.clusterlabs.org
>>>>>>>>
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>
>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
More information about the Pacemaker
mailing list