[Pacemaker] If 256 resources are load(ed), crmd will reboot.
Andrew Beekhof
andrew at beekhof.net
Thu May 29 01:00:03 EDT 2014
On 29 May 2014, at 12:28 pm, Yusuke Iida <yusk.iida at gmail.com> wrote:
> Hi, Andrew
>
> I'm sorry.
> It seems that the notation of the node name became another by syslog.
> In order to dispel misunderstanding, the report was newly acquired.
> I think that the signs are appearing in vm02/ha-log.
Got it :)
Ok, step 1 - stop logging debug.
Debug is accounting for 30% of the logs and all that writing to disk would be adding significantly to the cluster's workload.
Question: How have you got logging configured? Anything in /etc/sysconfig/pacemaker ?
I ask because pacemaker.log appears to have a jumble of syslog and regular file output:
May 29 10:45:26 vm02 cib[25603]: info: cib_perform_op: + /cib: @num_updates=1295
May 29 10:45:26 [25603] vm02 cib: info: cib_perform_op: + /cib: @num_updates=1295
Step 2 - can you try this patch:
diff --git a/crmd/te_callbacks.c b/crmd/te_callbacks.c
index 4d330a6..eba5f11 100644
--- a/crmd/te_callbacks.c
+++ b/crmd/te_callbacks.c
@@ -381,12 +381,15 @@ te_update_diff(const char *event, xmlNode * msg)
} else if(strstr(xpath, "/cib/configuration")) {
abort_transition(INFINITY, tg_restart, "Non-status change", change);
+ break; /* Wont be packaged with any resource operations we may be waiting for */
} else if(strstr(xpath, "/"XML_CIB_TAG_TICKETS) || safe_str_eq(name, XML_CIB_TAG_TICKETS)) {
abort_transition(INFINITY, tg_restart, "Ticket attribute change", change);
+ break; /* Wont be packaged with any resource operations we may be waiting for */
} else if(strstr(xpath, "/"XML_TAG_TRANSIENT_NODEATTRS"[") || safe_str_eq(name, XML_TAG_TRANSIENT_NODEATTRS)) {
abort_transition(INFINITY, tg_restart, "Transient attribute change", change);
+ break; /* Wont be packaged with any resource operations we may be waiting for */
} else if(strstr(xpath, "/"XML_LRM_TAG_RSC_OP"[") && safe_str_eq(op, "delete")) {
crm_action_t *cancel = NULL;
>
> May 29 10:43:37 vm02 crmd[25608]: error: config_query_callback:
> Local CIB query resulted in an error: Timer expired
> May 29 10:43:37 vm02 crmd[25608]: info: register_fsa_error_adv:
> Resetting the current action list
> May 29 10:43:37 vm02 crmd[25608]: error: do_log: FSA: Input I_ERROR
> from config_query_callback() received in state S_POLICY_ENGINE
> May 29 10:43:37 vm02 crmd[25608]: warning: do_state_transition: State
> transition S_POLICY_ENGINE -> S_RECOVERY [ input=I_ERROR
> cause=C_FSA_INTERNAL origin=config_query_callback ]
> May 29 10:43:37 vm02 crmd[25608]: warning: do_recover: Fast-tracking
> shutdown in response to errors
> May 29 10:43:37 vm02 crmd[25608]: warning: do_election_vote: Not
> voting in election, we're in state S_RECOVERY
>
> https://drive.google.com/file/d/0BwMFJItoO-fVSEd2MkRiOGxkelk/edit?usp=sharing
>
> Regards,
> Yusuke
>
> 2014-05-29 10:26 GMT+09:00 Andrew Beekhof <andrew at beekhof.net>:
>>
>> On 28 May 2014, at 6:42 pm, Yusuke Iida <yusk.iida at gmail.com> wrote:
>>
>>> Hi, Andrew
>>>
>>> I made the cluster load a setup to which 256 resources are started using crmsh.
>>> At this time, crmd changed into the S_RECOVERY state and rebooted.
>>>
>>> May 28 17:08:00 [14194] vm02 crmd: error:
>>> config_query_callback: Local CIB query resulted in an error: Timer
>>> expired
>>> May 28 17:08:00 [14194] vm02 crmd: info:
>>> register_fsa_error_adv: Resetting the current action list
>>> May 28 17:08:00 [14194] vm02 crmd: error: do_log: FSA: Input
>>> I_ERROR from config_query_callback() received in state S_POLICY_ENGINE
>>> May 28 17:08:00 [14194] vm02 crmd: warning:
>>> do_state_transition: State transition S_POLICY_ENGINE -> S_RECOVERY [
>>> input=I_ERROR cause=C_FSA_INTERNAL origin=config_query_callback ]
>>> May 28 17:08:00 [14194] vm02 crmd: warning: do_recover:
>>> Fast-tracking shutdown in response to errors
>>> May 28 17:08:00 [14194] vm02 crmd: warning: do_election_vote:
>>> Not voting in election, we're in state S_RECOVERY
>>>
>>> I think that query performed in large quantities cannot be processed.
>>> Before implementing cib_performance, abort_transition() was called only once.
>>>
>>> Is this corrected?
>>>
>>> report when a problem occurs is attached.
>>> https://drive.google.com/file/d/0BwMFJItoO-fVX0gxM1ptcE52WWs/edit?usp=sharing
>>
>> That doesn't appear to match the symptoms above.
>>
>>>
>>> Regards,
>>> Yusuke
>>> --
>>> ----------------------------------------
>>> METRO SYSTEMS CO., LTD
>>>
>>> Yusuke Iida
>>> Mail: yusk.iida at gmail.com
>>> ----------------------------------------
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
>
> --
> ----------------------------------------
> METRO SYSTEMS CO., LTD
>
> Yusuke Iida
> Mail: yusk.iida at gmail.com
> ----------------------------------------
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140529/c29e68f3/attachment-0003.sig>
More information about the Pacemaker
mailing list