[Pacemaker] 1.1.12: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)

[Andrew Beekhof]

On 31 Jul 2014, at 2:48 am, Cédric Dufour - Idiap Research Institute <cedric.dufour at idiap.ch> wrote:

> Hello,
> 
> After packaging pacemaker 1.1.12 for Debian/Wheezy (along corosync 1.4.6 and libqb 0.17.0), I have successfully initialized a new cluster.
> 
> The CIB processing improvement is amazing indeed! :-)
> 
> Back to a very simple test cluster, the only problem I have is with fencing, which fails altogether with "route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)" messages:
> 
> root at bc1hs22a01:~ # crm_mon -1
> Last updated: Wed Jul 30 18:40:11 2014
> Last change: Wed Jul 30 18:37:50 2014
> Stack: classic openais (with plugin)
> Current DC: bc1hs22a01 - partition with quorum
> Version: 1.1.12-561c4cf
> 2 Nodes configured, 2 expected votes
> 6 Resources configured
> Online: [ bc1hs22a01 bc1hs22a02 ]
> Full list of resources:
> Clone Set: NETWORK [NETWORK-ping]
>     Started: [ bc1hs22a01 bc1hs22a02 ]
> Clone Set: STONITH_bc1m1 [STONITH_bc1m1-P]
>     Started: [ bc1hs22a01 bc1hs22a02 ]
> Clone Set: STONITH_bc2m1 [STONITH_bc2m1-P]
>     Started: [ bc1hs22a01 bc1hs22a02 ]
> 
> root at bc1hs22a01:~ # stonith_admin -l bc1hs22a02
> STONITH_bc1m1-P
> STONITH_bc2m1-P:0
> STONITH_bc1m1-P:0
> STONITH_bc2m1-P
> 4 devices found
> 
> root at bc1hs22a01:~ # tail /var/log/corosync.rsyslog
> Jul 30 18:41:31 bc1hs22a01 stonith: [5363]: info: external/ibmbc device OK.
> Jul 30 18:41:31 bc1hs22a01 stonith: [5362]: info: external/ibmbc device OK.
> Jul 30 18:41:31 bc1hs22a01 stonith: [5361]: info: external/ibmbc device OK.
> Jul 30 18:41:31 bc1hs22a01 stonith: [5364]: info: external/ibmbc device OK.
> 
> root at bc1hs22a01:~ # stonith_admin -F bc1hs22a02
> 
> root at bc1hs22a01:~ # tail /var/log/corosync.rsyslog
> Jul 30 18:41:41 bc1hs22a01 stonith_admin[5411]:   notice: crm_log_args: Invoked: stonith_admin -F bc1hs22a02
> Jul 30 18:41:41 bc1hs22a01 stonithd[4754]:   notice: handle_request: Client stonith_admin.5411.fe1388ed wants to fence (off) 'bc1hs22a02' with device '(any)'
> Jul 30 18:41:41 bc1hs22a01 stonithd[4754]:   notice: initiate_remote_stonith_op: Initiating remote operation off for bc1hs22a02: 48b69f82-29ad-4c9a-af57-0e60ae5242e4 (0)
> Jul 30 18:41:41 bc1hs22a01 corosync[4686]:   [pcmk  ] WARN: route_ais_message: Sending message to local.stonith-ng failed: ipc delivery failed (rc=-2)

rc=-2 is coming from send_client_ipc(void *conn, const AIS_Message * ais_msg)

specifically:

    if (conn == NULL) {
        rc = -2;

So the plugin thinks that stonith-ng isn't connected.
More logs?

> 
> I've been trying to debug/troubleshoot this on my own but to no avail. :-(
> 
> What can be wrong ?
> Where/how should I look to understand where the problem lies ?
> 
> Thank you very much for your help,
> 
> Best,
> 
> Cédric
> 
> -- 
> 
> Cédric Dufour @ Idiap Research Institute
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140731/878268b6/attachment-0004.sig>