<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div>Booth version 0.1.0 has no retry method for packets, one single packet loss and the election does not work anymore.</div>
<div>Also it has a stupid checking of ballot values against promised values, in which case it communicates but does not do things right.</div>
<div>Booth version 0.1.0 is very sensitive about start order and saved ballot values as ticket attributes. The fault might come from your start procedure and the state it remembers from previous runs.</div>
<div>And it communicates through two different ways, one is the UDP port you configure in booth.conf used for renew and the other is a fixed TCP port somewhere in 23000 used for catchup.</div>
<div>I might give you a hint if you attach the complete log from <span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:10 </span>until <span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:08:10 for all three sites and attach the snipplet of booth.conf to see the order of lines with site and arbitrator.</span></div>
<div>Anyhow booth 0.1.0 is really unusable for productive environment, it has so many errors in it, you cant rely on it.</div>
<div>The booth version from GIT after 4th august 2014 behaves very well even in case of double fault (restart during network down) and builds under SLES and RHEL quite well now.</div>
<div>Rainer</div>
<div>
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Gesendet:</b> Dienstag, 26. August 2014 um 14:58 Uhr<br/>
<b>Von:</b> "Sutherland, Rob" <RSutherland@broadviewnet.com><br/>
<b>An:</b> "pacemaker@oss.clusterlabs.org" <pacemaker@oss.clusterlabs.org><br/>
<b>Betreff:</b> Re: [Pacemaker] SLES 11 SP3 boothd behaviour</div>
<div name="quoted-content"><!--p.MsoNormal, li.MsoNormal, div.MsoNormal {
margin: 0.0in;
font-size: 12.0pt;
font-family: "Times New Roman" , serif;
}
a:link, span.MsoHyperlink {
color: blue;
text-decoration: underline;
}
a:visited, span.MsoHyperlinkFollowed {
color: purple;
text-decoration: underline;
}
p {
margin: 0.0in;
font-size: 12.0pt;
font-family: "Times New Roman" , serif;
}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate {
margin: 0.0in;
font-size: 8.0pt;
font-family: Tahoma , sans-serif;
}
p.emailquote, li.emailquote, div.emailquote {
margin-top: 0.0in;
margin-right: 0.0in;
margin-bottom: 0.0in;
margin-left: 1.0pt;
border: none;
padding: 0.0in;
font-size: 12.0pt;
font-family: "Times New Roman" , serif;
}
span.EmailStyle19 {
font-family: Calibri , sans-serif;
color: windowtext;
}
span.BalloonTextChar {
font-family: Tahoma , sans-serif;
}
*.MsoChpDefault {
font-size: 10.0pt;
}
div.WordSection1 {
page: WordSection1;
}
-->
<div>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;">All nodes in question NTP from the same time source (yes, we have run into synchronicity issues in the past).</span></p>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;"> </span></p>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;">Interestingly, increasing the lease from 60 seconds to 120 seconds did not affect the behaviour.</span></p>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;"> </span></p>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;">Rob</span></p>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;"> </span></p>
<div>
<div style="border: none;border-top: solid rgb(181,196,223) 1.0pt;padding: 3.0pt 0.0in 0.0in 0.0in;">
<p class="MsoNormal"><b><span style="font-size: 10.0pt;font-family: Tahoma , sans-serif;">From:</span></b><span style="font-size: 10.0pt;font-family: Tahoma , sans-serif;"> John Lauro [mailto:john.lauro@covenanteyes.com]<br/>
<b>Sent:</b> Monday, August 25, 2014 6:17 PM<br/>
<b>To:</b> Sutherland, Rob<br/>
<b>Subject:</b> Re: [Pacemaker] SLES 11 SP3 boothd behaviour</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal" style="margin-bottom: 12.0pt;"><span style="color: black;">You probably already checked this, but just in case...<br/>
<br/>
No experience at all with geo-redundancy, but this sounds suspiciously like it could be a time sync problem. Have you tried something like "ntpq -np" on all 3 nodes and verify the offsets are all low (ie: < +/- 10) and times are in sync?<br/>
(Assuming you are running ntpd, and the process didn't stop.)</span><br/>
</p>
<div align="center" class="MsoNormal" style="text-align: center;">
<hr align="center" id="zwchr" size="2" width="100%"/></div>
<blockquote style="border: none;border-left: solid rgb(16,16,255) 1.5pt;padding: 0.0in 0.0in 0.0in 4.0pt;margin-left: 3.75pt;margin-top: 5.0pt;margin-bottom: 5.0pt;">
<p class="MsoNormal" style="margin-bottom: 12.0pt;"><b><span style="font-family: Helvetica , sans-serif;color: black;">From: </span></b><span style="font-family: Helvetica , sans-serif;color: black;">"Rob Sutherland" <<a href="RSutherland@BroadViewNet.com" target="_parent">RSutherland@BroadViewNet.com</a>><br/>
<b>To: </b><a href="pacemaker@oss.clusterlabs.org" target="_parent">pacemaker@oss.clusterlabs.org</a><br/>
<b>Sent: </b>Monday, August 25, 2014 3:43:34 PM<br/>
<b>Subject: </b>[Pacemaker] SLES 11 SP3 boothd behaviour</span></p>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Hello all,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">We’re in the process of implementing geo-redundancy on SLES 11 SP3 (version 0.1.0). We are seeing behavior in which site 2 in a geo-cluster decides that the ticket has expired long before actual expiry. Here’s an example time-line:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">1 - All sites (site 1, site 2 and arbitrator) agree on ticket owner and expiry. i.e. site 2 has the ticket with a 60-second expiry:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:10 linux-4i31 booth-arbitrator: [22526]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975690' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:10 bb5Btas0 booth-site: [27782]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975690' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:10 bb5Atas1 booth-site: [7826]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975690' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">2 - After 48 seconds (80% into lease), all three nodes are still in agreement:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Site 2: </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:58 bb5Btas0 booth-site: [27782]: info: command: 'crm_ticket -t geo-ticket -S owner -v 2' was executed </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:58 bb5Btas0 booth-site: [27782]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975738' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">The arbitrator: </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:58 linux-4i31 crm_ticket[23836]: notice: crm_log_args: Invoked: crm_ticket -t geo-ticket -S owner -v 2</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:58 linux-4i31 booth-arbitrator: [22526]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975738' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Site 1:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:58 bb5Atas1 booth-site: [7826]: info: command: 'crm_ticket -t geo-ticket -S owner -v 2' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:07:58 bb5Atas1 booth-site: [7826]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975738' was executed</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">3 - Site 2 decides that the ticket has expired (at the expiry time set in step 1)</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Aug 25 10:08:10 bb5Btas0 booth-site: [27782]: debug: lease expires ...</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">4 - At 10:08:58, both site 1 and the arbitrator expire the lease and pick a new master.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">I presume that there was some missed communication between site 2 and the rest of the geo-cluster. There is nothing in the logs to help debug this, though. Any hints on debugging this?</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">BTW: we only ever see this on a site 2 – never a site 1. This is consistent across several labs. Is there a bias towards site 1?</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Thanks in advance,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;">Rob</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size: 11.0pt;font-family: Calibri , sans-serif;color: black;"> </span></p>
</div>
<p class="MsoNormal"><br/>
<span style="font-family: Helvetica , sans-serif;color: black;">_______________________________________________<br/>
Pacemaker mailing list: <a href="Pacemaker@oss.clusterlabs.org" target="_parent">Pacemaker@oss.clusterlabs.org</a><br/>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br/>
<br/>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br/>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank"> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br/>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></span></p>
</blockquote>
<p class="MsoNormal"><span style="color: black;"> </span></p>
</div>
</div>
_______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a></div>
</div>
</div>
</div>
</div></div></body></html>