[Pacemaker] The larger cluster is tested.
yusuke iida
yusk.iida at gmail.com
Fri Nov 15 13:22:25 UTC 2013
Hi, Andrew
Thanks for the suggestion variety.
I fixed and tested the value of batch-limit by 1, 2, 3, and 4 from the
beginning, in order to confirm what batch-limit is suitable.
It was something like the following in my environment.
Timeout did not occur batch-limit=1 and 2.
batch-limit = 3 was 1 timeout.
batch-limit = 4 was 5 timeout.
I think the limit is still high in; From the above results, "limit =
QB_MAX (1, peers / 4)".
So I have created a fix to fixed to 2 batch-limit when it became a
state of extreme.
https://github.com/yuusuke/pacemaker/commit/efe2d6ebc55be39b8be43de38e7662f039b61dec
Results of the test several times, it seems to work without problems.
When batch-limit is fixed and tested, below has a report.
batch-limit=1
https://drive.google.com/file/d/0BwMFJItoO-fVNk8wTGlYNjNnSHc/edit?usp=sharing
batch-limit=2
https://drive.google.com/file/d/0BwMFJItoO-fVTnc4bXY2YXF2M2M/edit?usp=sharing
batch-limit=3
https://drive.google.com/file/d/0BwMFJItoO-fVYl9Gbks2VlJMR0k/edit?usp=sharing
batch-limit=4
https://drive.google.com/file/d/0BwMFJItoO-fVZnJIazd5MFQ1aGs/edit?usp=sharing
The report at the time of making it operate by my test code is the following.
https://drive.google.com/file/d/0BwMFJItoO-fVbzB0NjFLeVY3Zmc/edit?usp=sharing
Regards,
Yusuke
2013/11/13 Andrew Beekhof <andrew at beekhof.net>:
> Did you look at the load numbers in the logs?
> The CPUs are being slammed for over 20 minutes.
>
> The automatic tuning can only help so much, you're simply asking the cluster to do more work than it is capable of.
> Giving more priority to cib operations the come via IPC is one option, but as I explained earlier, it comes at the cost of correctness.
>
> Given the huge mismatch between the nodes' capacity and the tasks you're asking them to achieve, your best path forward is probably setting a load-threshold < 40% or a batch-limit <= 8.
> Or we could try a patch like the one below if we think that the defaults are not aggressive enough.
>
> diff --git a/crmd/throttle.c b/crmd/throttle.c
> index d77195a..7636d4a 100644
> --- a/crmd/throttle.c
> +++ b/crmd/throttle.c
> @@ -611,14 +611,14 @@ throttle_get_total_job_limit(int l)
> switch(r->mode) {
>
> case throttle_extreme:
> - if(limit == 0 || limit > peers/2) {
> - limit = peers/2;
> + if(limit == 0 || limit > peers/4) {
> + limit = QB_MAX(1, peers/4);
> }
> break;
>
> case throttle_high:
> - if(limit == 0 || limit > peers) {
> - limit = peers;
> + if(limit == 0 || limit > peers/2) {
> + limit = QB_MAX(1, peers/2);
> }
> break;
> default:
>
> This may also be worthwhile:
>
> diff --git a/crmd/throttle.c b/crmd/throttle.c
> index d77195a..586513a 100644
> --- a/crmd/throttle.c
> +++ b/crmd/throttle.c
> @@ -387,22 +387,36 @@ static bool throttle_io_load(float *load, unsigned int *blocked)
> }
>
> static enum throttle_state_e
> -throttle_handle_load(float load, const char *desc)
> +throttle_handle_load(float load, const char *desc, int cores)
> {
> - if(load > THROTTLE_FACTOR_HIGH * throttle_load_target) {
> + float adjusted_load = load;
> +
> + if(cores <= 0) {
> + /* No adjusting of the supplied load value */
> +
> + } else if(cores == 1) {
> + /* On a single core machine, a load of 1.0 is already too high */
> + adjusted_load = load * THROTTLE_FACTOR_MEDIUM;
> +
> + } else {
> + /* Normalize the load to be per-core */
> + adjusted_load = load / cores;
> + }
> +
> + if(adjusted_load > THROTTLE_FACTOR_HIGH * throttle_load_target) {
> crm_notice("High %s detected: %f", desc, load);
> return throttle_high;
>
> - } else if(load > THROTTLE_FACTOR_MEDIUM * throttle_load_target) {
> + } else if(adjusted_load > THROTTLE_FACTOR_MEDIUM * throttle_load_target) {
> crm_info("Moderate %s detected: %f", desc, load);
> return throttle_med;
>
> - } else if(load > THROTTLE_FACTOR_LOW * throttle_load_target) {
> + } else if(adjusted_load > THROTTLE_FACTOR_LOW * throttle_load_target) {
> crm_debug("Noticable %s detected: %f", desc, load);
> return throttle_low;
> }
>
> - crm_trace("Negligable %s detected: %f", desc, load);
> + crm_trace("Negligable %s detected: %f", desc, adjusted_load);
> return throttle_none;
> }
>
> @@ -464,22 +478,12 @@ throttle_mode(void)
> }
>
> if(throttle_load_avg(&load)) {
> - float simple = load / cores;
> - mode |= throttle_handle_load(simple, "CPU load");
> + mode |= throttle_handle_load(load, "CPU load", cores);
> }
>
> if(throttle_io_load(&load, &blocked)) {
> - float blocked_ratio = 0.0;
> -
> - mode |= throttle_handle_load(load, "IO load");
> -
> - if(cores) {
> - blocked_ratio = blocked / cores;
> - } else {
> - blocked_ratio = blocked;
> - }
> -
> - mode |= throttle_handle_load(blocked_ratio, "blocked IO ratio");
> + mode |= throttle_handle_load(load, "IO load", 0);
> + mode |= throttle_handle_load(blocked, "blocked IO ratio", cores);
> }
>
> if(mode & throttle_extreme) {
>
>
>
>
> On 12 Nov 2013, at 3:25 pm, yusuke iida <yusk.iida at gmail.com> wrote:
>
>> Hi, Andrew
>>
>> I'm sorry.
>> This report was a thing when two cores were assigned to the virtual machine.
>> https://drive.google.com/file/d/0BwMFJItoO-fVdlIwTVdFOGRkQ0U/edit?usp=sharing
>>
>> I'm sorry to be misleading.
>>
>> This is the report acquired with one core.
>> https://drive.google.com/file/d/0BwMFJItoO-fVSlo0dE0xMzNORGc/edit?usp=sharing
>>
>> It does not define the LRMD_MAX_CHILDREN on any node.
>> load-threshold is still default.
>> cib_max_cpu is set to 0.4 by the following processing.
>>
>> if(cores == 1) {
>> cib_max_cpu = 0.4;
>> }
>>
>> since -- if it exceeds 60%, it will be in the state of Extreme.
>> Nov 08 11:08:31 [2390] vm01 crmd: ( throttle.c:441 ) notice:
>> throttle_mode: Extreme CIB load detected: 0.670000
>>
>> From the state of a bit, DC is detecting that vm01 is in the state of Extreme.
>> Nov 08 11:08:32 [2387] vm13 crmd: ( throttle.c:701 ) debug:
>> throttle_update: Host vm01 supports a maximum of 2 jobs and
>> throttle mode 1000. New job limit is 1
>>
>> From the following log, a dynamic change of batch-limit also seems to
>> process satisfactorily.
>> # grep "throttle_get_total_job_limit" pacemaker.log
>> (snip)
>> Nov 08 11:08:31 [2387] vm13 crmd: ( throttle.c:629 ) trace:
>> throttle_get_total_job_limit: No change to batch-limit=0
>> Nov 08 11:08:32 [2387] vm13 crmd: ( throttle.c:632 ) trace:
>> throttle_get_total_job_limit: Using batch-limit=8
>> (snip)
>> Nov 08 11:10:32 [2387] vm13 crmd: ( throttle.c:632 ) trace:
>> throttle_get_total_job_limit: Using batch-limit=16
>>
>> The above shows that it is not solved even if it restricts the whole
>> number of jobs by batch-limit.
>> Are there any other methods of reducing a synchronous message?
>>
>> Internal IPC message is not so much.
>> Do not be able to handle even a little it on the way to handle the
>> synchronization message?
>>
>> Regards,
>> Yusuke
>>
>> 2013/11/12 Andrew Beekhof <andrew at beekhof.net>:
>>>
>>> On 11 Nov 2013, at 11:48 pm, yusuke iida <yusk.iida at gmail.com> wrote:
>>>
>>>> Execution of the graph was also checked.
>>>> Since the number of pending(s) is restricted to 16 from the middle, it
>>>> is judged that batch-limit is effective.
>>>> Observing here, even if a job is restricted by batch-limit, two or
>>>> more jobs are always fired(ed) in 1 second.
>>>> These performed jobs return a result and the synchronous message of
>>>> CIB generates them.
>>>> The node which continued receiving a synchronous message processes
>>>> there preferentially, and postpones an internal IPC message.
>>>> I think that it caused timeout.
>>>
>>> What load-threshold were you running this with?
>>>
>>> I see this in the logs:
>>> "Host vm10 supports a maximum of 4 jobs and throttle mode 0100. New job limit is 1"
>>>
>>> Have you set LRMD_MAX_CHILDREN=4 on these nodes?
>>> I wouldn't recommend that for a single core VM. I'd let the default of 2*cores be used.
>>>
>>>
>>> Also, I'm not seeing "Extreme CIB load detected". Are these still single core machines?
>>> If so it would suggest that something about:
>>>
>>> if(cores == 1) {
>>> cib_max_cpu = 0.4;
>>> }
>>> if(throttle_load_target > 0.0 && throttle_load_target < cib_max_cpu) {
>>> cib_max_cpu = throttle_load_target;
>>> }
>>>
>>> if(load > 1.5 * cib_max_cpu) {
>>> /* Can only happen on machines with a low number of cores */
>>> crm_notice("Extreme %s detected: %f", desc, load);
>>> mode |= throttle_extreme;
>>>
>>> is wrong.
>>>
>>> What was load-threshold configured as?
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>
>>
>>
>> --
>> ----------------------------------------
>> METRO SYSTEMS CO., LTD
>>
>> Yusuke Iida
>> Mail: yusk.iida at gmail.com
>> ----------------------------------------
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
--
----------------------------------------
METRO SYSTEMS CO., LTD
Yusuke Iida
Mail: yusk.iida at gmail.com
----------------------------------------
More information about the Pacemaker
mailing list