<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Comic Sans MS";
panose-1:3 15 7 2 3 3 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Comic Sans MS";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:136844358;
mso-list-type:hybrid;
mso-list-template-ids:-241015286 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1
{mso-list-id:1292590728;
mso-list-type:hybrid;
mso-list-template-ids:-1813622466 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l2
{mso-list-id:1785420330;
mso-list-type:hybrid;
mso-list-template-ids:1972803878 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l3
{mso-list-id:2097509301;
mso-list-type:hybrid;
mso-list-template-ids:1950752664 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>I have recently assumed the responsibility for maintaining code on one of my company’s products that uses Pacemaker/Heartbeat. I’m still coming up to speed on this code, and would like to solicit comments about some particular behavior. For reference, the Pacemaker version is 1.0.9.1, and Heartbeat is version 3.0.3.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>This product uses two host systems, each of which supports several disk enclosures, operating in an “active/passive” mode. The two hosts are connected by redundant, dedicated 10Gb Ethernet links, which are used for messaging between them. The disks in each enclosure are controlled by an instance of an application called SS. If an “active” host’s SS application fails for some reason, then the corresponding application on the “passive” host will assume control of the disks. Each application is assigned a Pacemaker resource, and the resource agent communicates with the appropriate SS instance. For reference, here’s a sample crm_mon output:<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>============<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Last updated: Tue Mar 5 06:10:22 2013<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Stack: Heartbeat<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Current DC: mgraid-12241530rn01433-0 (f4e5e15c-d06b-4e37-89b9-4621af05128f) - partition with quorum<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Version: 1.0.9-89bd754939df5150de7cd76835f98fe90851b677<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>2 Nodes configured, unknown expected votes<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>9 Resources configured.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>============<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Online: [ mgraid-12241530rn01433-0 mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Clone Set: Fencing<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Started: [ mgraid-12241530rn01433-0 mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Clone Set: cloneIcms<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Started: [ mgraid-12241530rn01433-0 mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Clone Set: cloneOmserver<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Started: [ mgraid-12241530rn01433-0 mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Master/Slave Set: ms-SS11451532RN01389<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Masters: [ mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Slaves: [ mgraid-12241530rn01433-0 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Master/Slave Set: ms-SS11481532RN01465<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Masters: [ mgraid-12241530rn01433-0 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Slaves: [ mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Master/Slave Set: ms-SS12171532RN01613<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Masters: [ mgraid-12241530rn01433-0 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Slaves: [ mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Master/Slave Set: ms-SS12241530RN01433<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Masters: [ mgraid-12241530rn01433-0 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Slaves: [ mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Master/Slave Set: ms-SS12391532RN01768<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Masters: [ mgraid-12241530rn01433-0 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Slaves: [ mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Master/Slave Set: ms-SS12391532RN01772<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Masters: [ mgraid-12241530rn01433-0 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Slaves: [ mgraid-12241530rn01433-1 ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>I’ve been investigating the system’s behavior when one or more master SS instances crashes, simulated by a </span><span style='font-family:"Courier New"'>kill</span><span style='font-family:"Comic Sans MS"'> command. I’ve noticed two behaviors of interest.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>First, in a simple case, where one master SS is </span><span style='font-family:"Courier New"'>killed</span><span style='font-family:"Comic Sans MS"'>, it takes about 10-12 seconds for the slave to complete the failover. From the log files, the DC issues the following notifications to the slave SS:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Pre_notify_demote<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Post_notify_demote<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Pre_notify_stop<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Post_notify_stop<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Pre_notify_promote<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Promote<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Post_notify_promote<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Monitor_3000<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Pre_notify_start<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l3 level1 lfo1'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Post_notify_start<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>These notifications and their confirmations appear to take about 1-2 seconds each, begging the following questions:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l2 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Is this sequence of notifications expected?<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l2 level1 lfo2'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Is the 10-12 second timeframe expected?<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>Second, in a more complex case, where the master SS for each instance is assigned to the same can, and each SS is in turn </span><span style='font-family:"Courier New"'>killed</span><span style='font-family:"Comic Sans MS"'> with an approximate 10-second delay between </span><span style='font-family:"Courier New"'>kill</span><span style='font-family:"Comic Sans MS"'> commands, there appear to be very long delays in processing the notifications. These delays appear to be associated with these factors<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo3'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>After an SS instance is </span><span style='font-family:"Courier New"'>killed</span><span style='font-family:"Comic Sans MS"'>, there’s a 10-second monitor notification which causes a new SS instance to be launched to replace the missing SS instance.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo3'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>It takes about 30 seconds for an SS instance to complete the startup process. The resource agent waits for that startup to complete before returning to </span><span style='font-family:"Courier New"'>crmd</span><span style='font-family:"Comic Sans MS"'>.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo3'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Until the resource agent returns, </span><span style='font-family:"Courier New"'>crmd</span><span style='font-family:"Comic Sans MS"'> does not process notifications for any other SS/resource.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>The net effect of these delays varies from one SS instance to another. In some cases, the “normal” failover occurs, taking 10-12 seconds. In other cases, there is no failover to the other host’s SS instance, and there is no master/active SS instance for 1-2 <b>minutes </b>(until an SS instance is re-launched following the </span><span style='font-family:"Courier New"'>kill</span><span style='font-family:"Comic Sans MS"'>), depending upon the number of disk enclosures and thus the number of SS instances.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>My first question in this case is simply whether the serialization of notifications among the various SS resources is expected? In other words, transition notifications for one resource are delayed until earlier notifications are completed. Is this the expected behavior? Secondly, once the SS instance has been restarted, there’s apparently no attempt to complete the failover; the new SS instance assumes the active/master role<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>Finally, a couple of general questions:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Is there any reason to believe that a later version of Pacemaker would behave differently?<b><o:p></o:p></b></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo4'><![if !supportLists]><span style='font-family:Symbol'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Comic Sans MS"'>Is there a mechanism by which the </span><span style='font-family:"Courier New"'>crmd</span><span style='font-family:"Comic Sans MS"'> (and </span><span style='font-family:"Courier New"'>lrmd</span><span style='font-family:"Comic Sans MS"'>) debug levels can be increased at run time (allowing more debug messages in the log output)?<b><o:p></o:p></b></span></p><p class=MsoNormal><b><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></b></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>Thanks very much for your help,<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'>Michael Powell<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Comic Sans MS"'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'><img width=160 height=50 id="Picture_x0020_1" src="cid:image001.gif@01CE1966.78F62940" alt=LogoSignature2><o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> Michael Powell<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> Staff Engineer<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> 15220 NW Greenbrier Pkwy<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> Suite 290<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> Beaverton, OR 97006<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> T 503-372-7327 M 503-789-3019 H 503-625-5332<o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:9.0pt;font-family:"Arial","sans-serif";color:#1F497D'> <a href="http://www.harmonicinc.com"><span style='color:blue'>www.harmonicinc.com</span></a><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>