<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
I performed some extra testing.<br>
<br>
I cleared my complete cib (cibadmin -E) and my crm_mon showed again
some information.<br>
<br>
I gradually started adding resources and monitored the cib.xml size
(cibadmin -Ql > cib.xml; ll -h cib.xml).<br>
This grew to about 430K. When adding another bunch of resources the
crm_mon didn't respond anymore.<br>
<br>
I checked the files in /dev/shm:<br>
-rw------- 1 hacluster root 516K jun 13 08:14
qb-cib_ro-event-58122-10738-13-data<br>
-rw------- 1 hacluster root 8,1K jun 13 08:14
qb-cib_ro-event-58122-10738-13-header<br>
-rw------- 1 hacluster root 516K jun 13 08:14
qb-cib_ro-request-58122-10738-13-data<br>
-rw------- 1 hacluster root 8,1K jun 13 08:14
qb-cib_ro-request-58122-10738-13-header<br>
-rw------- 1 hacluster root 516K jun 13 08:14
qb-cib_ro-response-58122-10738-13-data<br>
-rw------- 1 hacluster root 8,1K jun 13 08:14
qb-cib_ro-response-58122-10738-13-header<br>
-rw------- 1 hacluster root 96M jun 13 07:58
qb-cib_rw-event-58122-58130-12-data<br>
-rw------- 1 hacluster root 8,1K jun 13 07:58
qb-cib_rw-event-58122-58130-12-header<br>
-rw------- 1 hacluster root 96M jun 13 08:13
qb-cib_rw-request-58122-58130-12-data<br>
-rw------- 1 hacluster root 8,1K jun 13 07:58
qb-cib_rw-request-58122-58130-12-header<br>
-rw------- 1 hacluster root 96M jun 13 07:58
qb-cib_rw-response-58122-58130-12-data<br>
-rw------- 1 hacluster root 8,1K jun 13 07:58
qb-cib_rw-response-58122-58130-12-header<br>
<br>
Apparantly the cib_rw*data files are 96M size, but the cib_ro files
are only 516K.<br>
If this file must store the complete cib it is too small to store
our complete cib,<br>
which could explain why our crm_mon isn't working but the write
actions are giving no problems.<br>
<br>
gr.<br>
Johan<br>
<br>
<div class="moz-cite-prefix">On 13-06-14 09:35, Johan Huysmans
wrote:<br>
</div>
<blockquote cite="mid:539AA9A5.5050304@inuits.be" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hi,<br>
<br>
The PCMK_ipc_buffer was already set to 10000000 (10M).<br>
<br>
For testing I increased to buffer to 10000000 (100M), without
results.<br>
<br>
I decreased the buffer to the default 20480 (20K) as it then shows
the suggested value.<br>
If I leave it running for a couple of minutes I got these
suggested values:<br>
Jun 13 06:33:53 [62206] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (3854984 bytes suggested)<br>
Jun 13 06:33:54 [62206] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (7709968 bytes suggested)<br>
Jun 13 06:33:58 [62206] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (15419936 bytes suggested)<br>
Jun 13 06:35:55 [62206] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (30839872 bytes suggested)<br>
Jun 13 06:37:56 [62206] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (61679744 bytes suggested)<br>
Jun 13 07:16:47 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (3952852 bytes suggested)<br>
Jun 13 07:16:47 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (7905704 bytes suggested)<br>
Jun 13 07:18:03 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (15811408 bytes suggested)<br>
Jun 13 07:18:43 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (31622816 bytes suggested)<br>
Jun 13 07:18:48 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (63245632 bytes suggested)<br>
Jun 13 07:20:49 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (126491264 bytes suggested)<br>
Jun 13 07:22:50 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (252982528 bytes suggested)<br>
Jun 13 07:22:52 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (505965056 bytes suggested)<br>
Jun 13 07:23:57 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (1011930112 bytes suggested)<br>
Jun 13 07:24:51 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (2023860224 bytes suggested)<br>
Jun 13 07:26:52 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (-247246848 bytes suggested)<br>
Jun 13 07:27:22 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (-494493696 bytes suggested)<br>
Jun 13 07:29:22 [44112] SRV-5-1 cib: error:
crm_ipc_prepare: Could not compress the message into less than
the configured ipc limit (20480 bytes).Set PCMK_ipc_buffer to a
higher value (-988987392 bytes suggested)<br>
<br>
There is definitely something wrong. Is it the printing of the
suggested value or is it something else ?<br>
<br>
If I check the cib.xml files in /var/lib/pacemaker/cib/ all files
are a bit smaller then 300K.<br>
<br>
Changing these buffers did not solve my problem not getting
results from crm_mon.<br>
<br>
Gr.<br>
Johan<br>
<br>
<br>
<div class="moz-cite-prefix">On 13-06-14 01:13, Andrew Beekhof
wrote:<br>
</div>
<blockquote
cite="mid:931F9421-77B4-4B7E-A0EB-0D808963E88C@beekhof.net"
type="cite">
<pre wrap="">On 12 Jun 2014, at 10:53 pm, Johan Huysmans <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:johan.huysmans@inuits.be"><johan.huysmans@inuits.be></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi All,
I deployed Pacemaker 1.1.12-rc2 on our platform to test the cib changes.
This was needed on our setup as it contains 6 nodes, 150 resources and the cib process was using lots of cpu.
With a limited set of resources (6 nodes, 30 resources) everything worked as expected, including crm_mon.
When loading the complete set of resources we lost the crm_mon functionality on all nodes.
The cluster is running as expected (running all resources) however we don't have any visibility.
I noticed that operations performing changes did actually work like (crm resource stop <resourcename>),
but crm resource status didn't work (using crmsh-2.0+git46-1.1.x86_64).
I noticed that /dev/shm/qb-cib_ro* files are created, and lsof shows that they are both opened by crm_mon and cib.
When executing "crm_mon -1" I get following messages in /var/log/messages (and /var/log/pacemaker.log)
Jun 12 12:47:38 [8062] SRV-5-1 cib: notice: crm_ipcs_sendv: Response 2 to 0x1810370[17836] (1091618 bytes) failed: Resource temporarily unavailable (-11)
Jun 12 12:47:38 [8062] SRV-5-1 cib: warning: do_local_notify: Sync reply to crm_mon failed: No message of desired type
Restarting the pacemaker and cman service of 1 node didn't solve it.
What is causing this problem and how can I resolve it ?
</pre>
</blockquote>
<pre wrap="">Almost certainly you're hitting IPC limits associated with large clusters.
You should be able to tune:
# PCMK_ipc_buffer=20480
In /etc/sysconfig/pacemaker and then restart the cluster.
Note also:
# For non-systemd based systems, prefix 'export' to each enabled line
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Pacemaker mailing list: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a>
Project Home: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a>
Getting started: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
Bugs: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Pacemaker mailing list: <a class="moz-txt-link-abbreviated" href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a>
<a class="moz-txt-link-freetext" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a>
Project Home: <a class="moz-txt-link-freetext" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a>
Getting started: <a class="moz-txt-link-freetext" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
Bugs: <a class="moz-txt-link-freetext" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a>
</pre>
</blockquote>
<br>
</body>
</html>