[ClusterLabs] Connection to the CPG API failed: Library error (2)

[Karthikeyan Ramasamy]

Hello Pacemaker Users,
  We are having a problem with cman failing, when the host server is under load and processing traffic.  Following information is available in the logs,  


Jan 30 06:53:13 [2041] vmc0135      attrd:    error: pcmk_cpg_dispatch:         Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2043] vmc0135       crmd:    error: pcmk_cpg_dispatch:         Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2043] vmc0135       crmd:    error: crmd_cs_destroy:   connection terminated
Jan 30 06:53:13 [2041] vmc0135      attrd:     crit: attrd_cs_destroy:  Lost connection to Corosync service!
Jan 30 06:53:13 [2043] vmc0135       crmd:     info: qb_ipcs_us_withdraw:       withdrawing server sockets
Jan 30 06:53:13 [2038] vmc0135        cib:     info: crm_cs_flush:      Sent 0 CPG messages  (1 remaining, last=129): Library error (2)
Jan 30 06:53:13 [2038] vmc0135        cib:    error: pcmk_cpg_dispatch:         Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2038] vmc0135        cib:    error: cib_cs_destroy:    Corosync connection lost!  Exiting.
Jan 30 06:53:13 [2038] vmc0135        cib:     info: terminate_cib:     cib_cs_destroy: Exiting fast...
Jan 30 06:53:13 [2038] vmc0135        cib:     info: crm_client_destroy:        Destroying 0 events
Jan 30 06:53:13 [2041] vmc0135      attrd:   notice: main:      Exiting...
Jan 30 06:53:13 [2041] vmc0135      attrd:   notice: main:      Disconnecting client 0x611c30, pid=2043...
Jan 30 06:53:13 [2038] vmc0135        cib:     info: qb_ipcs_us_withdraw:       withdrawing server sockets
Jan 30 06:53:13 [2038] vmc0135        cib:     info: crm_client_destroy:        Destroying 0 events
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:    error: pcmk_cpg_dispatch:         Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2043] vmc0135       crmd:     info: tengine_stonith_connection_destroy:        Fencing daemon disconnected
Jan 30 06:53:13 [2043] vmc0135       crmd:   notice: crmd_exit:         Forcing immediate exit: Link has been severed (67)
Jan 30 06:53:13 [2043] vmc0135       crmd:     info: crm_xml_cleanup:   Cleaning up memory from libxml2
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:    error: stonith_peer_cs_destroy:   Corosync connection terminated
Jan 30 06:53:13 [2041] vmc0135      attrd:    error: attrd_cib_connection_destroy:      Connection to the CIB terminated...
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:    error: cfg_connection_destroy:    Connection destroyed
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:     info: stonith_shutdown:  Terminating with  1 clients
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:   notice: pcmk_shutdown_worker:      Shuting down Pacemaker
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:   notice: stop_child:        Stopping crmd: Sent -15 to process 2043
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:    error: pcmk_cpg_dispatch:         Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:    error: mcp_cpg_destroy:   Connection destroyed
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:     info: crm_xml_cleanup:   Cleaning up memory from libxml2
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:     info: cib_connection_destroy:    Connection to the CIB closed.
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:     info: crm_client_destroy:        Destroying 0 events
Jan 30 06:53:13 [2038] vmc0135        cib:     info: crm_client_destroy:        Destroying 0 events
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:     info: qb_ipcs_us_withdraw:       withdrawing server sockets
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:     info: main:      Done
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:     info: crm_xml_cleanup:   Cleaning up memory from libxml2
Jan 30 06:53:13 [2038] vmc0135        cib:     info: qb_ipcs_us_withdraw:       withdrawing server sockets
Jan 30 06:53:13 [2038] vmc0135        cib:     info: crm_client_destroy:        Destroying 0 events
Jan 30 06:53:13 [2038] vmc0135        cib:     info: qb_ipcs_us_withdraw:       withdrawing server sockets
Jan 30 06:53:13 [2038] vmc0135        cib:     info: crm_xml_cleanup:   Cleaning up memory from libxml2


We are trying to figure out when is this happening.  In the meantime, guidance from expert users would be very helpful for us.  Please let me know if you have already seen this issue and the probable causes.

Thanks,
Karthik.