<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:'times new roman', 'new york', times, serif;font-size:12pt"><div style="color: black; font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">Andrew,</div><div style="color: black; font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><br></div><div style="color: black; font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; ">Comments at end with <BS></div><div><br><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; "><font size="2" face="Tahoma"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Andrew Beekhof <andrew@beekhof.net><br><b><span style="font-weight: bold;">To:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight: bold;">Cc:</span></b> The Pacemaker cluster resource manager
<pacemaker@oss.clusterlabs.org><br><b><span style="font-weight: bold;">Sent:</span></b> Fri, April 15, 2011 4:28:52 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [Pacemaker] Question regarding starting of master/slave resources and ELECTIONs<br></font><br>
On Fri, Apr 15, 2011 at 5:58 AM, Bob Schatz <<a ymailto="mailto:bschatz@yahoo.com" href="mailto:bschatz@yahoo.com">bschatz@yahoo.com</a>> wrote:<br>> Andrew,<br>> Thanks for the help<br>> Comments inline with <BS><br>> ________________________________<br>> From: Andrew Beekhof <<a ymailto="mailto:andrew@beekhof.net" href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>><br>> To: Bob Schatz <<a ymailto="mailto:bschatz@yahoo.com" href="mailto:bschatz@yahoo.com">bschatz@yahoo.com</a>><br>> Cc: The Pacemaker cluster resource manager <<a ymailto="mailto:pacemaker@oss.clusterlabs.org" href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a>><br>> Sent: Thu, April 14, 2011 2:14:40 AM<br>> Subject: Re: [Pacemaker] Question regarding starting of master/slave<br>> resources and ELECTIONs<br>><br>> On Thu, Apr 14, 2011 at 10:49 AM, Andrew Beekhof <<a
ymailto="mailto:andrew@beekhof.net" href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>> wrote:<br>><br>>>>> I noticed that 4 of the master/slave resources will start right away but<br>>>>> the<br>>>>> 5 master/slave resource seems to take a minute or so and I am only<br>>>>> running<br>>>>> with one node.<br>>>>> Is this expected?<br>>>><br>>>> Probably, if the other 4 take around a minute each to start.<br>>>> There is an lrmd config variable that controls how much parallelism it<br>>>> allows (but i forget the name).<br>>>> <Bob> It's max-children and I set it to 40 for this test to see if it<br>>>> would<br>>>> change the behavior. (/sbin/lrmadmin -p max-children 40)<br>>><br>>> Thats surprising. I'll have a look at the logs.<br>><br>> Looking at the logs, I see a couple
of things:<br>><br>><br>> This is very bad:<br>> Apr 12 19:33:42 mgraid-S000030311-1 crmd: [17529]: WARN: get_uuid:<br>> Could not calculate UUID for mgraid-s000030311-0<br>> Apr 12 19:33:42 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> populate_cib_nodes_ha: Node mgraid-s000030311-0: no uuid found<br>><br>> For some reason pacemaker cant get the node's uuid from heartbeat.<br>><br>> <BS> I create the uuid when the node comes up.<br><br>Heartbeat should have already created it before pacemaker even got<br>started though.<br><br>><br>> So we start a few things:<br>><br>> Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>> do_lrm_rsc_op: Performing<br>> key=23:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>> op=SSS000030311:0_start_0 )<br>> Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>> do_lrm_rsc_op: Performing<br>>
key=49:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>> op=SSJ000030312:0_start_0 )<br>> Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>> do_lrm_rsc_op: Performing<br>> key=75:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>> op=SSJ000030313:0_start_0 )<br>> Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>> do_lrm_rsc_op: Performing<br>> key=101:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>> op=SSJ000030314:0_start_0 )<br>><br>> But then another change comes in:<br>><br>> Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>> abort_transition_graph: need_abort:59 - Triggered transition abort<br>> (complete=0) : Non-status change<br>><br>> Normally we'd recompute and keep going, but it was a(nother) replace<br>> operation, so:<br>><br>> Apr 12 19:33:42 mgraid-S000030311-1 crmd: [17529]: info:<br>> do_state_transition: State transition S_TRANSITION_ENGINE -><br>>
S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL<br>> origin=do_cib_replaced ]<br>><br>> All the time goes here:<br>><br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Timer popped (timeout=20000,<br>> abort_level=1000000, complete=true)<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Ignoring timeout while not in transition<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Timer popped (timeout=20000,<br>> abort_level=1000000, complete=true)<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Ignoring timeout while not in transition<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Timer popped (timeout=20000,<br>> abort_level=1000000, complete=true)<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd:
[17529]: WARN:<br>> action_timer_callback: Ignoring timeout while not in transition<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Timer popped (timeout=20000,<br>> abort_level=1000000, complete=true)<br>> Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>> action_timer_callback: Ignoring timeout while not in transition<br>> Apr 12 19:37:00 mgraid-S000030311-1 crmd: [17529]: ERROR:<br>> crm_timer_popped: Integration Timer (I_INTEGRATED) just popped!<br>><br>> but its not at all clear to me why - although certainly avoiding the<br>> election would help.<br>> Is there any chance to load all the changes at once?<br>><br>> <BS> Yes. That worked. I created the configuration in a file and then did<br>> a "crm configure load update <filename>" to avoid the election<br>> Possibly the delay related to the UUID issue above, possibly it
might<br>> be related to one of these two patches that went in after 1.0.9<br>><br>> andrew (stable-1.0) High: crmd: Make sure we always poke the FSA after<br>> a transition to clear any TE_HALT actions CS: 9187c0506fd3 On:<br>> 2010-07-07<br>> andrew (stable-1.0) High: crmd: Reschedule the PE_START action if its<br>> not already running when we try to use it CS: e44dfe49e448 On:<br>> 2010-11-11<br>><br>> Could you try turning on debug and/or a more recent version?<br>><br>> <BS> I turned on debug and grabbed the logs, configuration and and<br>> /var/lib/pengine directory. They are attached.<br>> Unfortunately I cannot try a new version with this hardware at this<br>> time. :(<br><br>Based on the new logs, it looks like the problem is resolved by<br>loading everything once.<br>All start actions appear to occur within a second of Apr 14
20:35:43.<br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; "><BS> This run did not create the resources all at once. It did start the resources at Apr 14 20:35:43.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; ">However, when looking at the log debug/ctlr0/ha-debug you will see these entries (Note this node is not the pengine):</div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; "><br></div><div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52 mgraid-S000030311-0 lrmd: [15228]: debug: on_msg_get_state:state of rsc mgraid-stonith:0 is LRM_RSC_IDLE</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52
mgraid-S000030311-0 lrmd: [15228]: debug: on_msg_get_state:state of rsc SSJ000030313:0 is LRM_RSC_BUSY</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52 mgraid-S000030311-0 crmd: [15231]: WARN: msg_to_op(1324): failed to get the value of field lrm_opstatus from a ha_msg</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52 mgraid-S000030311-0 crmd: [15231]: info: msg_to_op: Message follows:</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52 mgraid-S000030311-0 crmd: [15231]: info: MSG: Dumping message with 16 fields</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52 mgraid-S000030311-0 crmd: [15231]: info: MSG[0] : [lrm_t=op]</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14
20:35:52 mgraid-S000030311-0 crmd: [15231]: info: MSG[1] : [lrm_rid=SSJ000030313:0]</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="2">Apr 14 20:35:52 mgraid-S000030311-0 crmd: [15231]: info: MSG[2] : [lrm_op=start]</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; "><br></div></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; ">This was reported as bug report 2580.</div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; "><br></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; ">Later, this resource was stopped for some reason by pengine.</div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; "><br></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; ">Unfortunately, I grabbed
the logs before the resources such as <span class="Apple-style-span" style="font-size: small; ">SSJ000030313:0 were stopped. :( :(</span></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; "><span class="Apple-style-span" style="font-size: small; "><br></span></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; "><span class="Apple-style-span" style="font-size: small; ">Anyway, I was able to get things to work correctly as summarized below.</span></div><div style="color: black; font-family: arial, helvetica, sans-serif; font-size: 13px; "><span class="Apple-style-span" style="font-size: small; "><br></span></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">I think that 2580 covers all of the issues to get the problem fixed. Do you agree?</font></div><div style="color: black; font-family: arial,
helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">In the mean time, a summary of the work around was that I did these steps:</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">1. Increase the number of threads lrmd has to fork off the shell scripts to do start, monitor, etc with this command in /etc/ha.d/resource.d/startstop (post-start action)</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">
# /sbin/lrmadmin -p max-children 8</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">2. Create all of the resources needed in one file (basically take the output from a "crm configure show" on an already configured system and copy to a file) as opposed to creating each resource and associated linkage with individual crm commands. This avoids the "replace" step which may happen in crm. The "replace" operation causes the crmds to do an election which causes me problems. (I assume it is a bug related to 2580 but I am not sure)</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica,
sans-serif; "><font class="Apple-style-span" size="2">3. Do an "update" as opposed to a "replace" of the HA configuration with this step:</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"> # crm configure load update <filename></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">Thanks Andrew for your help!</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font
class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2">Bob</font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"> </font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="color: black; font-family: arial, helvetica, sans-serif; "><font class="Apple-style-span" size="2"><br></font></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 13px; color: black; "><br></div></div><div style="position: fixed; color: black; font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "></div>
</div></body></html>