[Pacemaker] Cannot create more than 27 multistate resources

Wed Jul 9 08:49:20 UTC 2014

Hi,

[root at vsanqa11 ~]# rpm -qa | grep pcs ; rpm -qa | grep pace ; rpm -qa |
grep libqb; rpm -qa | grep coro; rpm -qa | grep cman
pcs-0.9.90-2.el6.centos.2.noarch
pacemaker-cli-1.1.10-14.el6_5.3.x86_64
pacemaker-libs-1.1.10-14.el6_5.3.x86_64
pacemaker-1.1.10-14.el6_5.3.x86_64
pacemaker-cluster-libs-1.1.10-14.el6_5.3.x86_64
libqb-devel-0.16.0-2.el6.x86_64
libqb-0.16.0-2.el6.x86_64
corosynclib-1.4.1-17.el6_5.1.x86_64
corosync-1.4.1-17.el6_5.1.x86_64
cman-3.0.12.1-59.el6_5.2.x86_64
[root at vsanqa11 ~]# uname -a
Linux vsanqa11 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux
[root at vsanqa11 ~]# cat /etc/redhat-release
CentOS release 6.3 (Final)

Created 27 resources

[root at vsanqa11 ~]# pcs status
Cluster name: vsanqa11_12
Last updated: Wed Jul  9 01:30:05 2014
Last change: Wed Jul  9 01:24:12 2014 via cibadmin on vsanqa11
Stack: cman
Current DC: vsanqa11 - partition with quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
54 Resources configured

Online: [ vsanqa11 vsanqa12 ]

Full list of resources:

 Master/Slave Set: ms-0feba438-0f16-4de5-bf18-9d576cf4dd26
[vha-0feba438-0f16-4de5-bf18-9d576cf4dd26]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-329ad1bd-2f9c-483d-a052-270731aefd70
[vha-329ad1bd-2f9c-483d-a052-270731aefd70]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-b5b9e8dc-87c9-4229-b425-870d1bc1f107
[vha-b5b9e8dc-87c9-4229-b425-870d1bc1f107]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-56112cc8-4c56-4454-84ea-ba9b7b82dde7
[vha-56112cc8-4c56-4454-84ea-ba9b7b82dde7]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-20fd5ed6-8e72-4a6b-9b1a-73bd96e1f253
[vha-20fd5ed6-8e72-4a6b-9b1a-73bd96e1f253]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-77af4d7e-d78c-4799-a24a-7536d2cccc25
[vha-77af4d7e-d78c-4799-a24a-7536d2cccc25]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-913f6e7e-932f-4c3f-9e7f-a70c4153b4c7
[vha-913f6e7e-932f-4c3f-9e7f-a70c4153b4c7]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-3d3ae48c-1955-4c64-b951-f2a2621a70b7
[vha-3d3ae48c-1955-4c64-b951-f2a2621a70b7]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-a8101da1-636f-483b-90a9-9a18fd5a5793
[vha-a8101da1-636f-483b-90a9-9a18fd5a5793]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-290e0fdd-19c6-4b40-8046-0fa2c94a7320
[vha-290e0fdd-19c6-4b40-8046-0fa2c94a7320]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-8341368b-6b6a-4bf0-bb0a-38b32ffec2f4
[vha-8341368b-6b6a-4bf0-bb0a-38b32ffec2f4]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-6695f479-0d21-4388-9540-a440c32e0944
[vha-6695f479-0d21-4388-9540-a440c32e0944]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-85fe517e-9a2b-4ac1-b6e1-e4da57b49969
[vha-85fe517e-9a2b-4ac1-b6e1-e4da57b49969]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-5b9adf5c-3bb5-4353-8c5f-a3e1508f7c4a
[vha-5b9adf5c-3bb5-4353-8c5f-a3e1508f7c4a]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-57b4b7a1-d3bb-4b94-b8dc-8a84e4a3de03
[vha-57b4b7a1-d3bb-4b94-b8dc-8a84e4a3de03]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-b4fbac14-ef19-4861-93e5-14574e101484
[vha-b4fbac14-ef19-4861-93e5-14574e101484]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-ec4836d3-b768-4abd-83bf-3a61143346ce
[vha-ec4836d3-b768-4abd-83bf-3a61143346ce]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-28d6db1a-bdba-4fc0-b74c-ef74548cf714
[vha-28d6db1a-bdba-4fc0-b74c-ef74548cf714]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-c6cfd538-17f4-4c4b-b259-731e2cac75f3
[vha-c6cfd538-17f4-4c4b-b259-731e2cac75f3]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-06d19988-7549-4882-b780-fd598714ec7f
[vha-06d19988-7549-4882-b780-fd598714ec7f]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-ec105b52-42b1-42f5-931f-249fbc2f16c4
[vha-ec105b52-42b1-42f5-931f-249fbc2f16c4]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-6cc9a97b-9714-4cab-bf52-103c46b8593f
[vha-6cc9a97b-9714-4cab-bf52-103c46b8593f]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-b80b4a69-67e2-43bd-8eba-f6c18f78d706
[vha-b80b4a69-67e2-43bd-8eba-f6c18f78d706]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-24817d3c-98e0-4408-9bce-67d8bb2495db
[vha-24817d3c-98e0-4408-9bce-67d8bb2495db]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-a6d08422-8592-42cf-995a-8c72c6368cd2
[vha-a6d08422-8592-42cf-995a-8c72c6368cd2]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-100f3ec9-03aa-476c-9569-60463cc2ac8e
[vha-100f3ec9-03aa-476c-9569-60463cc2ac8e]
     Masters: [ vsanqa12 ]
     Slaves: [ vsanqa11 ]
 Master/Slave Set: ms-7ec2decb-5426-4ecf-a60f-9996ea0ca445
[vha-7ec2decb-5426-4ecf-a60f-9996ea0ca445]
     Masters: [ vsanqa11 ]
     Slaves: [ vsanqa12 ]

[root at vsanqa11 ~]# pcs resource create
vha-3de5ab16-9917-4b90-93d2-7b04fc71879c ocf:heartbeat:vgc-cm-agent.ocf
cluster_uuid=3de5ab16-9917-4b90-93d2-7b04fc71879c op monitor role="Master"
interval=30s timeout=100s op monitor role="Master" interval=30s timeout=100s

pcs status output includes
 vha-3de5ab16-9917-4b90-93d2-7b04fc71879c
(ocf::heartbeat:vgc-cm-agent.ocf):      Started vsanqa11

[root at vsanqa11 ~]# pcs resource master
ms-3de5ab16-9917-4b90-93d2-7b04fc71879c
vha-3de5ab16-9917-4b90-93d2-7b04fc71879c meta clone-max=2
globally-unique=false target-role=started
Error: unable to locate command: /usr/sbin/cibadmin

Deleted one of the resources that was created successfully and reran above
command to create master resource ms-3de5ab16-9917-4b90-93d2-7b04fc71879c
and it worked.

Tried this multiple times and each time saw the same behavior

Attached is the debug output for the above command. This output contains
string "unable to locate command: /usr/sbin/cibadmin"

Regards,
kiran
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140709/5f8a1691/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pcs_debug
Type: application/octet-stream
Size: 264699 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140709/5f8a1691/attachment-0003.obj>