[Pacemaker] Pacemaker 1.1.12 cib testing, crm_mon doesn't work

[Andrew Beekhof]

On 12 Jun 2014, at 10:53 pm, Johan Huysmans <johan.huysmans at inuits.be> wrote:

> Hi All,
> 
> I deployed Pacemaker 1.1.12-rc2 on our platform to test the cib changes.
> This was needed on our setup as it contains 6 nodes, 150 resources and the cib process was using lots of cpu.
> 
> With a limited set of resources (6 nodes, 30 resources) everything worked as expected, including crm_mon.
> When loading the complete set of resources we lost the crm_mon functionality on all nodes.
> The cluster is running as expected (running all resources) however we don't have any visibility.
> 
> I noticed that operations performing changes did actually work like (crm resource stop <resourcename>),
> but crm resource status didn't work (using crmsh-2.0+git46-1.1.x86_64).
> 
> I noticed that /dev/shm/qb-cib_ro* files are created, and lsof shows that they are both opened by crm_mon and cib.
> 
> 
> When executing "crm_mon -1" I get following messages in /var/log/messages (and /var/log/pacemaker.log)
> Jun 12 12:47:38 [8062] SRV-5-1        cib:   notice: crm_ipcs_sendv:     Response 2 to 0x1810370[17836] (1091618 bytes) failed: Resource temporarily unavailable (-11)
> Jun 12 12:47:38 [8062] SRV-5-1        cib:  warning: do_local_notify:     Sync reply to crm_mon failed: No message of desired type
> 
> 
> Restarting the pacemaker and cman service of 1 node didn't solve it.
> 
> 
> What is causing this problem and how can I resolve it ?

Almost certainly you're hitting IPC limits associated with large clusters.

You should be able to tune:

# PCMK_ipc_buffer=20480

In /etc/sysconfig/pacemaker and then restart the cluster.

Note also:

# For non-systemd based systems, prefix 'export' to each enabled line


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140613/ad90f891/attachment-0004.sig>