I figured it out, turns out there are some undocumented properties for ping. What was happening was it was using the default monitor timeout of 20 seconds, but it was killing the ping process after this time, but the ping wasn't finished yet.<div>
<br></div><div>See:</div><div><br></div><div><a href="http://hg.clusterlabs.org/pacemaker/stable-1.0/raw-file/tip/extra/resources/ping">http://hg.clusterlabs.org/pacemaker/stable-1.0/raw-file/tip/extra/resources/ping</a><br>
<br>Anlu<br><br><div class="gmail_quote">On Fri, Feb 17, 2012 at 11:09 AM, Anlu Wang <span dir="ltr"><<a href="mailto:anlu@mixpanel.com">anlu@mixpanel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also, in my constraints section, for the ping connectivity resource location definitions, a node attribute is not specified on rsc_location. What is the default value of node then?<span class="HOEnZb"><font color="#888888"><div>
<br></div></font></span><div><span class="HOEnZb"><font color="#888888">Anlu</font></span><div><div class="h5"><br><br><div class="gmail_quote">
On Fri, Feb 17, 2012 at 10:57 AM, Anlu Wang <span dir="ltr"><<a href="mailto:anlu@mixpanel.com" target="_blank">anlu@mixpanel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm running 1.0.8. In accordance with the bug in the post you linked, I changed the config so that interval is greater than dampen. Here is the relevant section now:<div><br></div><div><div><font face="'courier new', monospace"> <clone id="connectivity_resource"></font></div>
<div><font face="'courier new', monospace"> <primitive class="ocf" id="ping" provider="pacemaker" type="ping"></font></div><div><font face="'courier new', monospace"> <instance_attributes id="ping-attrs"></font></div>
<div><font face="'courier new', monospace"> <nvpair id="pingd-dampen" name="dampen" value="5s"/></font></div><div><font face="'courier new', monospace"> <nvpair id="pingd-multiplier" name="multiplier" value="100"/></font></div>
<div><font face="'courier new', monospace"> <nvpair id="pingd-hosts" name="host_list" value="10.54.130.6 10.54.130.8 10.54.130.7 50.97.196.101 50.97.196.103 <a href="tel:50.97.196.102" value="+15097196102" target="_blank">50.97.196.102</a>"/></font></div>
<div><font face="'courier new', monospace"> </instance_attributes></font></div><div><font face="'courier new', monospace"> <operations></font></div><div><font face="'courier new', monospace"> <op id="ping-monitor-10s" interval="10s" name="monitor" timeout="60s"/></font></div>
<div><font face="'courier new', monospace"> </operations></font></div><div><font face="'courier new', monospace"> </primitive></font></div><div><font face="'courier new', monospace"> <meta_attributes id="connectivity_resource-meta_attributes"></font></div>
<div><font face="'courier new', monospace"> <nvpair id="connectivity_resource-meta_attributes-target-role" name="target-role" value="Started"/></font></div><div><font face="'courier new', monospace"> </meta_attributes></font></div>
<div><font face="'courier new', monospace"> </clone></font></div><div><br></div><div>The scores are still not what I expect however, and when I disable the internal interface on a node, nothing happens with failover.</div>
<div><br></div><div>Also, I've noticed this in my syslog:</div><div><br></div><div><div>Feb 17 06:26:11 anlutest2 lrmd: [1137]: WARN: ping:1:monitor process (PID 9380) timed out (try 1). Killing with signal SIGTERM (15).</div>
<div>Feb 17 06:26:11 anlutest2 lrmd: [1137]: info: RA output: (ping:1:monitor:stderr) Terminated</div><div>Feb 17 06:26:11 anlutest2 ping[9380]: [15745]: INFO: They use TERM to bring us down. No such luck.</div><div>Feb 17 06:26:11 anlutest2 ping[9380]: [15747]: ERROR: Unexpected result for 'ping -n -q -W 3 -c 5 <a href="tel:50.97.196.103" value="+15097196103" target="_blank">50.97.196.103</a>' 143: </div>
<div><br></div><div>So it looks like the ping command is failing for some reason, but when I run it manually, it succeeds...</div><div><br></div><div>Really at a loss here, any help is appreciated!</div><span><font color="#888888"><div>
<br></div><div>
Anlu</div></font></span><div><div><br><div class="gmail_quote">On Fri, Feb 17, 2012 at 3:26 AM, Dejan Muhamedagic <span dir="ltr"><<a href="mailto:dejanmm@fastmail.fm" target="_blank">dejanmm@fastmail.fm</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<div><br>
On Thu, Feb 16, 2012 at 07:57:14PM -0800, Anlu Wang wrote:<br>
> I have three machines named anlutest1, anlutest2, and anlutest3 that I'm<br>
> trying to get IP failover working on. I'm using heartbeat for the messaging<br>
> layer, and everything works great when a machine goes down. But I also<br>
> would like to failover an IP when EITHER the eth0 or eth1 network<br>
> interfaces fail. From reading<br>
><br>
> <a href="http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ch09s03s03.html" target="_blank">http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ch09s03s03.html</a><br>
><br>
> it seems the right way to do this is to add a ping resource.<br>
><br>
> Here is my XML configuration:<br>
><br>
> <a href="http://pastebin.com/05z7eB2s" target="_blank">http://pastebin.com/05z7eB2s</a><br>
<br>
</div>The configuration seems OK, though obviously monitors are<br>
scheduled back-to-back (the postponed operations messages below).<br>
I guess that you should increase the intervals or reduce the<br>
dampen period. Which version of Pacemaker do you run? Perhaps<br>
also take a look at this thread:<br>
<br>
<a href="http://oss.clusterlabs.org/pipermail/pacemaker/2011-April/009942.html" target="_blank">http://oss.clusterlabs.org/pipermail/pacemaker/2011-April/009942.html</a><br>
<br>
Thanks,<br>
<br>
Dejan<br>
<div><div><br>
> This config doesn't work for me. Using the showscores.sh script found at:<br>
><br>
> <a href="http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg00410.html" target="_blank">http://www.mail-archive.com/pacemaker@oss.clusterlabs.org/msg00410.html</a><br>
><br>
> I see that my scores are:<br>
><br>
> Resource Score Node Stickiness #Fail<br>
> Migration-Threshold<br>
> address01 0 anlutest3 0 0<br>
><br>
> address01 1006 anlutest1 0 5<br>
><br>
> address01 50 anlutest2 0 157<br>
><br>
> address02 0 anlutest3 0 0<br>
><br>
> address02 1050 anlutest2 0 2<br>
><br>
> address02 6 anlutest1 0 0<br>
><br>
> address03 1000 anlutest3 0 7<br>
><br>
> address03 50 anlutest2 0<br>
><br>
> address03 6 anlutest1 0 0<br>
><br>
> ping:0 0 anlutest1 0 6<br>
><br>
> ping:0 0 anlutest2 0 14<br>
><br>
> ping:0 0 anlutest3 0 0<br>
><br>
> ping:1 0 anlutest2 0<br>
><br>
> ping:1 0 anlutest3 0 28<br>
><br>
> ping:1 -1000000 anlutest1 0 0<br>
><br>
> ping:2 0 anlutest3 0 13<br>
><br>
> ping:2 -1000000 anlutest1 0 0<br>
><br>
> ping:2 -1000000 anlutest2 0<br>
><br>
> which make no sense at all. I don't see how I could be getting these scores<br>
> of 50 and 1006. When I take down an interface on anlutest3, I see scores of<br>
> 4 and 1004, which sort of make sense, just the multiplier of 100 isn't<br>
> working. I was experimenting with changing values, so maybe its caching old<br>
> values. If so, how do I enforce the new values?<br>
><br>
> Furthermore, shouldn't there be no scores of 0? If all 6 IPs I am pinging<br>
> return successfully, shouldn't my scores be either 600 or 1600?<br>
><br>
> In my syslog I also see a ton of messages like<br>
><br>
> Feb 17 03:54:47 anlutest2 lrmd: [1137]: info: perform_op:2877: operations<br>
> on resource address01 already delayed<br>
> Feb 17 03:54:48 anlutest2 lrmd: [1137]: info: perform_op:2873: operation<br>
> monitor[419] on ocf::ping::ping:1 for client 1140, its parameters:<br>
> CRM_meta_clone=[1] host_list=[10.54.130.6 10.54.130.8 10.54.130.7<br>
> <a href="tel:50.97.196.101" value="+15097196101" target="_blank">50.97.196.101</a> <a href="tel:50.97.196.103" value="+15097196103" target="_blank">50.97.196.103</a> 50.9CRM_meta_clone_max=[3] dampen=[60s]<br>
> crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false] multiplier=[10000]<br>
> CRM_meta_name=[monitor] CRM_meta_timeout=[60000] CRM_meta_interval=[5000]<br>
> for rsc is already running.<br>
> Feb 17 03:54:48 anlutest2 lrmd: [1137]: info: perform_op:2883: postponing<br>
> all ops on resource ping:1 by 1000 ms<br>
> Feb 17 03:54:48 anlutest2 lrmd: [1137]: info: perform_op:2873: operation<br>
> monitor[171] on ocf::ping::ping:2 for client 1140, its parameters:<br>
> CRM_meta_clone=[2] host_list=[10.54.130.6 10.54.130.8 10.54.130.7<br>
> <a href="tel:50.97.196.101" value="+15097196101" target="_blank">50.97.196.101</a> <a href="tel:50.97.196.103" value="+15097196103" target="_blank">50.97.196.103</a> 50.9CRM_meta_clone_max=[3] dampen=[60s]<br>
> crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false] multiplier=[1]<br>
> CRM_meta_name=[monitor] CRM_meta_timeout=[30000] CRM_meta_interval=[5000]<br>
> for rsc is already running.<br>
> Feb 17 03:54:48 anlutest2 lrmd: [1137]: info: perform_op:2883: postponing<br>
> all ops on resource ping:2 by 1000 ms<br>
><br>
> and occasionally<br>
><br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: info: attrd_trigger_update:<br>
> Sending flush op to all hosts for: pingd (4000)<br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: info: attrd_ha_callback: flush<br>
> message from anlutest2<br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: WARN: find_nvpair_attr: Multiple<br>
> attributes match name=pingd<br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: info: find_nvpair_attr: Value:<br>
> 50 #011(id=status-d619a94e-ebba-4ed0-8e0f-89837dd7506b-pingd)<br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: info: find_nvpair_attr: Value: 3<br>
> #011(id=status-ab3c1a25-9471-48f7-9c0b-c76238abd402-pingd)<br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: info: attrd_perform_update: Sent<br>
> update -40: pingd=4000<br>
> Feb 17 03:54:33 anlutest2 attrd: [1139]: ERROR: attrd_cib_callback: Update<br>
> -40 for pingd=4000 failed: Required data for this CIB API call not found<br>
><br>
> Could someone just take a look at my config and let me know what I'm doing<br>
> wrong? Or if there's a better way to do what I want to do...<br>
><br>
> Thanks in advance,<br>
> Anlu<br>
<br>
</div></div>> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>