[Pacemaker] abrupt power failure problem

[Bernd Schubert]

On Tuesday 15 June 2010, Schaefer, Diane E wrote:
> Hi,
>   We are having trouble with our two node cluster after one node
>  experiences an abrupt power failure.  The resources do not seem to start
>  on the remaining node (ie DRBD resources do not promote to master).  In
>  the log we notice:
> 
> Jan  8 02:12:27 qpr4 stonithd: [6622]: info: external_run_cmd: Calling
>  '/usr/lib64/stonith/plugins/external/ipmi reset qpr3' returned 256 Jan  8
>  02:12:27 qpr4 stonithd: [6622]: CRIT: external_reset_req: 'ipmi reset' for
>  host qpr3 failed with rc 256 Jan  8 02:12:27 qpr4 stonithd: [5854]: info:
>  failed to STONITH node qpr3 with local device stonith0 (exitcode 5), gonna
>  try the next local device Jan  8 02:12:27 qpr4 stonithd: [5854]: info: we
>  can't manage qpr3, broadcast request to other nodes Jan  8 02:13:27 qpr4
>  stonithd: [5854]: ERROR: Failed to STONITH the node qpr3: optype=RESET,
>  op_result=TIMEOUT
> 
> Jan  8 02:13:27 qpr4 stonithd: [6763]: info: external_run_cmd: Calling
>  '/usr/lib64/stonith/plugins/external/ipmi reset qpr3' returned 256 Jan  8
>  02:13:27 qpr4 stonithd: [6763]: CRIT: external_reset_req: 'ipmi reset' for
>  host qpr3 failed with rc 256 Jan  8 02:13:27 qpr4 stonithd: [5854]: info:
>  failed to STONITH node qpr3 with local device stonith0 (exitcode 5), gonna
>  try the next local device Jan  8 02:13:27 qpr4 stonithd: [5854]: info: we
>  can't manage qpr3, broadcast request to other nodes Jan  8 02:14:27 qpr4
>  stonithd: [5854]: ERROR: Failed to STONITH the node qpr3: optype=RESET,
>  op_result=TIMEOUT

Without looking at your hb_report, this already looks pretty clear - this node 
tries to reset the other node using IPMI and that fails, of course, as the 
node to be reset is powered off.
When we had that problem in the past, we simply temporarily removed the failed 
node from the pacemaker configuration: crm node remove <node-name>


Cheers,
Bernd