[Pacemaker] 1.1.12: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)

[Cédric Dufour - Idiap Research Institute]

Hello,

After packaging pacemaker 1.1.12 for Debian/Wheezy (along corosync 1.4.6 and libqb 0.17.0), I have successfully initialized a new cluster.

The CIB processing improvement is amazing indeed! :-)

Back to a very simple test cluster, the only problem I have is with fencing, which fails altogether with "route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)" messages:

root at bc1hs22a01:~ # crm_mon -1
Last updated: Wed Jul 30 18:40:11 2014
Last change: Wed Jul 30 18:37:50 2014
Stack: classic openais (with plugin)
Current DC: bc1hs22a01 - partition with quorum
Version: 1.1.12-561c4cf
2 Nodes configured, 2 expected votes
6 Resources configured
Online: [ bc1hs22a01 bc1hs22a02 ]
Full list of resources:
 Clone Set: NETWORK [NETWORK-ping]
     Started: [ bc1hs22a01 bc1hs22a02 ]
 Clone Set: STONITH_bc1m1 [STONITH_bc1m1-P]
     Started: [ bc1hs22a01 bc1hs22a02 ]
 Clone Set: STONITH_bc2m1 [STONITH_bc2m1-P]
     Started: [ bc1hs22a01 bc1hs22a02 ]

root at bc1hs22a01:~ # stonith_admin -l bc1hs22a02
 STONITH_bc1m1-P
 STONITH_bc2m1-P:0
 STONITH_bc1m1-P:0
 STONITH_bc2m1-P
4 devices found

root at bc1hs22a01:~ # tail /var/log/corosync.rsyslog
Jul 30 18:41:31 bc1hs22a01 stonith: [5363]: info: external/ibmbc device OK.
Jul 30 18:41:31 bc1hs22a01 stonith: [5362]: info: external/ibmbc device OK.
Jul 30 18:41:31 bc1hs22a01 stonith: [5361]: info: external/ibmbc device OK.
Jul 30 18:41:31 bc1hs22a01 stonith: [5364]: info: external/ibmbc device OK.

root at bc1hs22a01:~ # stonith_admin -F bc1hs22a02

root at bc1hs22a01:~ # tail /var/log/corosync.rsyslog
Jul 30 18:41:41 bc1hs22a01 stonith_admin[5411]:   notice: crm_log_args: Invoked: stonith_admin -F bc1hs22a02
Jul 30 18:41:41 bc1hs22a01 stonithd[4754]:   notice: handle_request: Client stonith_admin.5411.fe1388ed wants to fence (off) 'bc1hs22a02' with device '(any)'
Jul 30 18:41:41 bc1hs22a01 stonithd[4754]:   notice: initiate_remote_stonith_op: Initiating remote operation off for bc1hs22a02: 48b69f82-29ad-4c9a-af57-0e60ae5242e4 (0)
Jul 30 18:41:41 bc1hs22a01 corosync[4686]:   [pcmk  ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)

I've been trying to debug/troubleshoot this on my own but to no avail. :-(

What can be wrong ?
Where/how should I look to understand where the problem lies ?

Thank you very much for your help,

Best,

Cédric

-- 

Cédric Dufour @ Idiap Research Institute