<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:10pt"><div><span>Hi Folks,</span></div><div><span><br></span></div><div><span style="font-size: 10pt; ">The problem with my configuration issue was resolved. In short, my</span><span style="font-size: 10pt; "> configuration </span><span style="font-size: 10pt; ">did not allow for a location where the stonith resource could run. Adding an appropriate 'location' entry allowed the stonith resource to run, and then stonith immediately worked as expected.</span></div><div><span><br></span></div><div><span>No, I did not resolve this issue; instead I enlisted support from </span><span style="font-size: 10pt; ">Andreas Kurz & friends at Hastexo.com. </span><span style="font-size: 10pt; ">I received the type of support that was needed; it was very fast; it was very professional; most importantly, I would *certainly*
enlist their support again. Thank you Andreas !</span></div><div> </div><div><span style="font-size: 10px; font-family: arial, helvetica, sans-serif; ">Brett Lee<br>Everything Penguin - <span class="Apple-tab-span" style="white-space:pre;"></span><a rel="nofollow" target="_blank" href="http://etpenguin.com/">http://etpenguin.com</a><br></span></div><div><br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; margin-top: 5px; padding-left: 5px;"> <div style="font-size: 10pt; font-family: arial, helvetica, sans-serif; "> <div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "> <div dir="ltr"> <font size="2" face="Arial"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Dejan Muhamedagic <dejanmm@fastmail.fm><br> <b><span style="font-weight: bold;">To:</span></b> Brett Lee <brettlee@yahoo.com>; The Pacemaker cluster resource manager
<pacemaker@oss.clusterlabs.org> <br> <b><span style="font-weight: bold;">Sent:</span></b> Monday, July 2, 2012 7:27 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Pacemaker] newb - stonith not working - require others to stonith node<br> </font> </div> <br>Hi,<br><br>On Fri, Jun 29, 2012 at 08:43:24AM -0700, Brett Lee wrote:<br>> Hi - <br>> <br>> <br>> Am new to pacemaker and now have a shiny new configuration that will not stonith. This is a test system using KVM and external/libvirt - all VMs are running CentOS 5.<br>> <br>> Am (really) hoping someone might be willing to help troubleshoot this configuration. Thank you for your time and effort!<br>> <br>> <br>> <br>> The items that are suspect to me are:<br>> 1. st-nodes has no 'location' entry<br>> 2. logs report node_list=<br>> 3. resource st-nodes is Stopped<br>> <br>> Have attached a clip of the
configuration below. The full configuration and log file may be found at - <a href="http://pastebin.com/bS87FXUr" target="_blank">http://pastebin.com/bS87FXUr</a><br>> <br>> Per 'stonith -t external/libvirt -h' I have configured stonith using:<br><br>Did you try fencing manually with this stonith program? You can<br>do it like this:<br><br>stonith -t external/libvirt hostlist="st15-mds1,st15-mds2,st15-oss1,st15-oss2" hypervisor_uri="qemu+ssh://wc0008/system" -T reset st15-mds1<br><br>> primitive st-nodes stonith:external/libvirt \<br>> params hostlist="st15-mds1,st15-mds2,st15-oss1,st15-oss2" hypervisor_uri="qemu+ssh://wc0008/system" stonith-timeout="30" \<br><br>I'm not sure if ',' works here as a separator, better use a<br>space.<br><br>stonith-timeout is effectively ignored here. Use the cluster<br>property for that.<br><br>> op start
interval="0" timeout="60"<br>> \<br>> op stop interval="0" timeout="60" \<br>> op monitor interval="60"<br>> <br>> And a section of the log file is:<br>> <br>> Jun 29 11:02:07 st15-mds2 stonithd: [4485]: ERROR: Failed to STONITH the node st15-mds1: optype=RESET, op_result=TIMEOUT<br><br>This indicates that fencing was attempted. But it timed out.<br>Perhaps take a look at the libvirt logs?<br><br>Thanks,<br><br>Dejan<br><br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: tengine_stonith_callback: call=-65, optype=1, node_name=st15-mds1, result=2, node_list=, action=23:90:0:aac961e7-b06b-4dfd-ae60-c882407b16b5<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: ERROR: tengine_stonith_callback: Stonith of st15-mds1 failed (2)... aborting transition.<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: abort_transition_graph:
tengine_stonith_callback:409 - Triggered transition abort (complete=0) : Stonith failed<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: update_abort_priority: Abort<br>> action done superceeded by restart<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: run_graph: ====================================================<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: notice: run_graph: Transition 90 (Complete=2, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pengine/pe-warn-173.bz2): Stopped<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: te_graph_trigger: Transition 90 is now complete<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<br>> Jun 29 11:02:07
st15-mds2 crmd: [4490]: info: do_state_transition: All 3 cluster nodes are eligible to run resources.<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_pe_invoke: Query 299: Requesting the current CIB: S_POLICY_ENGINE<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_pe_invoke_callback: Invoking the PE: query=299,<br>> ref=pe_calc-dc-1340982127-223, seq=396, quorate=1<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status: Node st15-mds2 is online<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: pe_fence_node: Node st15-mds1 will be fenced because it is un-expectedly down<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status_fencing: ha_state=active, ccm_state=false, crm_state=online, join_state=member,
expected=member<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: determine_online_status: Node st15-mds1 is unclean<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status: Node st15-oss1 is online<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: determine_online_status: Node st15-oss2 is online<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice:<br>> native_print: lustre-OST0000 (ocf::heartbeat:Filesystem): Started st15-oss1<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-OST0001 (ocf::heartbeat:Filesystem): Started st15-oss1<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-OST0002 (ocf::heartbeat:Filesystem): Started st15-oss2<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print:
lustre-OST0003 (ocf::heartbeat:Filesystem): Started st15-oss2<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: lustre-MDT0000 (ocf::heartbeat:Filesystem): Started st15-mds1<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: native_print: st-nodes (stonith:external/libvirt): Stopped <br>> Jun 29 11:02:07 st15-mds2 pengine:<br>> [4489]: info: native_color: Resource st-nodes cannot run anywhere<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: custom_action: Action lustre-MDT0000_stop_0 on st15-mds1 is unrunnable (offline)<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: custom_action: Marking node st15-mds1 unclean<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000 on st15-mds2<br>> Jun 29 11:02:07 st15-mds2
pengine: [4489]: WARN: stage6: Scheduling Node st15-mds1 for STONITH<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: native_stop_constraints: lustre-MDT0000_stop_0 is implicit after st15-mds1 is fenced<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave resource lustre-OST0000 (Started st15-oss1)<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave resource lustre-OST0001 (Started st15-oss1)<br>> Jun<br>> 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave resource lustre-OST0002 (Started st15-oss2)<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave resource lustre-OST0003 (Started st15-oss2)<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Move resource lustre-MDT0000
(Started st15-mds1 -> st15-mds2)<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: notice: LogActions: Leave resource st-nodes (Stopped)<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: WARN: process_pe_message: Transition 91: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-174.bz2<br>> Jun<br>> 29 11:02:07 st15-mds2 crmd: [4490]: info: unpack_graph: Unpacked transition 91: 7 actions in 7 synapses<br>> Jun 29 11:02:07 st15-mds2 pengine: [4489]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: do_te_invoke: Processing graph 91
(ref=pe_calc-dc-1340982127-223) derived from /var/lib/pengine/pe-warn-174.bz2<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: te_pseudo_action: Pseudo action 21 fired and confirmed<br>> Jun 29 11:02:07 st15-mds2 crmd: [4490]: info: te_fence_node: Executing reboot fencing operation (23) on st15-mds1 (timeout=60000)<br>> Jun 29 11:02:07 st15-mds2 stonithd: [4485]: info: client tengine [pid: 4490] requests a STONITH operation RESET on node st15-mds1<br>> Jun 29 11:02:07 st15-mds2 stonithd: [4485]: info: we can't manage st15-mds1, broadcast request to other nodes<br>> Jun 29 11:02:07 st15-mds2 stonithd:<br>> [4485]: info: Broadcasting the message succeeded: require others to stonith node st15-mds1.<br>> <br>> Thank you!<br>> <br>> <br>> Brett Lee<br>> Everything Penguin - <a href="http://etpenguin.com/" target="_blank">http://etpenguin.com</a><br><br>>
_______________________________________________<br>> Pacemaker mailing list: <a ymailto="mailto:Pacemaker@oss.clusterlabs.org" href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>> <br>> Project Home: <a href="http://www.clusterlabs.org/" target="_blank">http://www.clusterlabs.org</a><br>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>> Bugs: <a href="http://bugs.clusterlabs.org/" target="_blank">http://bugs.clusterlabs.org</a><br><br><br><br> </div> </div> </blockquote></div> </div></body></html>