[Pacemaker] newb - stonith not working - require others to stonith node

Dejan Muhamedagic dejanmm at fastmail.fm
Mon Jul 2 13:27:11 UTC 2012 

Hi,

On Fri, Jun 29, 2012 at 08:43:24AM -0700, Brett Lee wrote:
> Hi - 
> 
> 
> Am new to pacemaker and now have a shiny new configuration that will not stonith.  This is a test system using KVM and external/libvirt - all VMs are running CentOS 5.
> 
> Am (really) hoping someone might be willing to help troubleshoot this configuration.  Thank you for your time and effort!
> 
> 
> 
> The items that are suspect to me are:
> 1.  st-nodes has no 'location' entry
> 2.  logs report node_list=
> 3.  resource st-nodes is Stopped
> 
> Have attached a clip of the configuration below.  The full configuration and log file may be found at - http://pastebin.com/bS87FXUr
> 
> Per 'stonith -t external/libvirt -h' I have configured stonith using:

Did you try fencing manually with this stonith program? You can
do it like this:

stonith -t external/libvirt hostlist="st15-mds1,st15-mds2,st15-oss1,st15-oss2" hypervisor_uri="qemu+ssh://wc0008/system" -T reset st15-mds1

> primitive st-nodes stonith:external/libvirt \
>         params hostlist="st15-mds1,st15-mds2,st15-oss1,st15-oss2" hypervisor_uri="qemu+ssh://wc0008/system" stonith-timeout="30" \

I'm not sure if ',' works here as a separator, better use a
space.

stonith-timeout is effectively ignored here. Use the cluster
property for that.

>         op start interval="0" timeout="60"
>  \
>         op stop interval="0" timeout="60" \
>         op monitor interval="60"
> 
> And a section of the log file is:
> 
> Jun 29 11:02:07 st15-mds2 stonithd: [4485]: ERROR: Failed to STONITH the node st15-mds1: optype=RESET, op_result=TIMEOUT

This indicates that fencing was attempted. But it timed out.
Perhaps take a look at the libvirt logs?

Thanks,

Dejan

> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: tengine_stonith_callback: call=-65, optype=1, node_name=st15-mds1, result=2, node_list=, action=23:90:0:aac961e7-b06b-4dfd-ae60-c882407b16b5
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: ERROR: tengine_stonith_callback: Stonith of st15-mds1 failed (2)... aborting transition.
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: abort_transition_graph: tengine_stonith_callback:409 - Triggered transition abort (complete=0) : Stonith failed
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: update_abort_priority: Abort
>  action done superceeded by restart
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: run_graph: ====================================================
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: notice: run_graph: Transition 90 (Complete=2, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pengine/pe-warn-173.bz2): Stopped
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: te_graph_trigger: Transition 90 is now complete
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_state_transition: All 3 cluster nodes are eligible to run resources.
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_pe_invoke: Query 299: Requesting the current CIB: S_POLICY_ENGINE
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_pe_invoke_callback: Invoking the PE: query=299,
>  ref=pe_calc-dc-1340982127-223, seq=396, quorate=1
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status: Node st15-mds2 is online
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: pe_fence_node: Node st15-mds1 will be fenced because it is un-expectedly down
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status_fencing:     ha_state=active, ccm_state=false, crm_state=online, join_state=member, expected=member
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: determine_online_status: Node st15-mds1 is unclean
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status: Node st15-oss1 is online
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status: Node st15-oss2 is online
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice:
>  native_print: lustre-OST0000    (ocf::heartbeat:Filesystem):    Started st15-oss1
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-OST0001    (ocf::heartbeat:Filesystem):    Started st15-oss1
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-OST0002    (ocf::heartbeat:Filesystem):    Started st15-oss2
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-OST0003    (ocf::heartbeat:Filesystem):    Started st15-oss2
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-MDT0000    (ocf::heartbeat:Filesystem):    Started st15-mds1
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: st-nodes    (stonith:external/libvirt):    Stopped 
> Jun 29 11:02:07 st15-mds2 pengine:
>  [4489]: info: native_color: Resource st-nodes cannot run anywhere
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: custom_action: Action lustre-MDT0000_stop_0 on st15-mds1 is unrunnable (offline)
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: custom_action: Marking node st15-mds1 unclean
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: RecurringOp:  Start recurring monitor (120s) for lustre-MDT0000 on st15-mds2
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: stage6: Scheduling Node st15-mds1 for STONITH
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: native_stop_constraints: lustre-MDT0000_stop_0 is implicit after st15-mds1 is fenced
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave   resource lustre-OST0000    (Started st15-oss1)
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave   resource lustre-OST0001    (Started st15-oss1)
> Jun
>  29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave   resource lustre-OST0002    (Started st15-oss2)
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave   resource lustre-OST0003    (Started st15-oss2)
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Move    resource lustre-MDT0000    (Started st15-mds1 -> st15-mds2)
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave   resource st-nodes    (Stopped)
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: process_pe_message: Transition 91: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-174.bz2
> Jun
>  29 11:02:07 st15-mds2 crmd: [4490]: info: unpack_graph: Unpacked transition 91: 7 actions in 7 synapses
> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: process_pe_message: Configuration WARNINGs found during PE processing.  Please run "crm_verify -L" to identify issues.
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_te_invoke: Processing graph 91 (ref=pe_calc-dc-1340982127-223) derived from /var/lib/pengine/pe-warn-174.bz2
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: te_pseudo_action: Pseudo action 21 fired and confirmed
> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: te_fence_node: Executing reboot fencing operation (23) on st15-mds1 (timeout=60000)
> Jun 29 11:02:07 st15-mds2 stonithd: [4485]: info: client tengine [pid: 4490] requests a STONITH operation RESET on node st15-mds1
> Jun 29 11:02:07 st15-mds2 stonithd: [4485]: info: we can't manage st15-mds1, broadcast request to other nodes
> Jun 29 11:02:07 st15-mds2 stonithd:
>  [4485]: info: Broadcasting the message succeeded: require others to stonith node st15-mds1.
> 
> Thank you!
> 
>  
> Brett Lee
> Everything Penguin - http://etpenguin.com

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list