[Pacemaker] resource moving unnecessarily due to ping race condition

Brad Johnson bjohnson at ecessa.com
Tue Sep 27 08:40:07 EDT 2011 

The patch alone does not give an advantage to the active node. But 
remember I said we are using an fping resource agent we wrote that 
varies the dampening based on which node it is running on and whether 
the score is rising or falling. But the dampening it sets was being 
over-ridden by the attrd flush message, since it stops the timer and 
sends the score immediately. That RA, along with the patch, solves our 
problem. I can now reboot ping nodes all day long without our resource 
failing back and forth, while still allowing legitimate fail-overs when 
a node truly has better network connectivity than the other.


On 09/26/2011 09:57 PM, Andrew Beekhof wrote:
> On Mon, Sep 26, 2011 at 10:57 PM, Brad Johnson<bjohnson at ecessa.com>  wrote:
>> I agree that the patch assumes the use of "pingd" for the attribute name,
>> and there may be a better way of coding that. However, I don't see how
>> setting dampen=0 fixes our problem.  The problem occurs when a ping node
>> becomes inaccessible to all nodes in the cluster (it is rebooted for
>> example). Without giving any timing advantage to the currently active node,
> The patch doesn't do this though.
>
>> it is essentially just a race between the nodes to see who notices the
>> outage first and can update the attribute fastest.
> Now I'm confused.
>
> You said "we do not want the other node to be able to challenge us to
> an immediate score comparison".
> dampen=0 does the same thing as the patch... it tells attrd to update
> the CIB immediately, without waiting to give everyone a chance to
> notice the change in connectivity too.
>
>> The result is we see
>> fail-over when the ping node goes down, and fail-back when it comes back up.
>> The fact is that dampening alone does not solve this. Which is why we use a
>> resource agent that uses selective dampening based on where the resource is
>> running.
>>
>> On 09/25/2011 08:58 PM, Andrew Beekhof wrote:
>>> On Fri, Sep 23, 2011 at 9:53 PM, Brad Johnson<bjohnson at ecessa.com>    wrote:
>>>> Yes, but the patch only affects the pingd attribute.
>>> Use of the name 'pingd' isnt mandatory though.
>>>
>>>> And we do not want the
>>>> other node to be able to challenge us to an immediate score comparison.
>>>> That
>>>> is the whole idea behind the fping OCF resource agent we are using, to
>>>> give
>>>> the timing advantage to the node currently running the resource by
>>>> delaying
>>>> rising scores on the idle, and falling scores on the active node.
>>> Why not just set dampen=0?
>>>
>>>> On 09/22/2011 09:10 PM, Andrew Beekhof wrote:
>>>>> On Tue, Sep 20, 2011 at 10:34 PM, Brad Johnson<bjohnson at ecessa.com>
>>>>>   wrote:
>>>>>> It is not necessarily the case that the outside world can't reach the
>>>>>> cluster. Ours is a multi-homed device connecting to multiple WANs and
>>>>>> LANs.
>>>>>> We want the device with the best connectivity to be the active device.
>>>>>> To
>>>>>> get around the problem of failovers occurring when a ping node reboots
>>>>>> for
>>>>>> example, I have written an fping OCF RA that uses different dampening
>>>>>> delays
>>>>>> based on if it is running on the active or idle device. I have also
>>>>>> patched
>>>>>> pacemaker attrd.c to fix it so it doesn't send an immediate update when
>>>>>> it
>>>>>> receives a flush message from the other node. This was causing it to
>>>>>> ignore
>>>>>> any running delay timer.
>>>>> Thats the point of the flush message though.  So that all nodes write
>>>>> their current value at the same time.
>>>>>
>>>>>> Here is that patch:
>>>>>>
>>>>>> --- tools/attrd.orig.c    2011-09-13 08:29:46.946820348 -0500
>>>>>> +++ tools/attrd.c    2011-09-14 13:33:59.606894754 -0500
>>>>>> @@ -348,10 +348,14 @@
>>>>>>          attrd_local_callback(xml);
>>>>>>
>>>>>>      } else if(ignore == NULL || safe_str_neq(from, attrd_uname)) {
>>>>>> +        const char *attr  = crm_element_value(xml, F_ATTRD_ATTRIBUTE);
>>>>>> +        /* Don't send update for score if msg is from other node */
>>>>>> +        if(safe_str_eq(from, attrd_uname) || safe_str_neq(attr,
>>>>>> "pingd")) {
>>>>>>          crm_info("%s message from %s", op, from);
>>>>>>          hash_entry = find_hash_entry(xml);
>>>>>>          stop_attrd_timer(hash_entry);
>>>>>>          attrd_perform_update(hash_entry);
>>>>>> +        }
>>>>>>      }
>>>>>>      free_xml(xml);
>>>>>>   }
>>>>>>
>>>>>>
>>>>>> On 09/19/2011 10:51 PM, Andrew Beekhof wrote:
>>>>>>> On Sun, Sep 11, 2011 at 2:30 AM, Vadym Chepkov<vchepkov at gmail.com>
>>>>>>>   wrote:
>>>>>>>> On Sep 8, 2011, at 3:40 PM, Florian Haas wrote:
>>>>>>>>
>>>>>>>>>>> On 09/08/11 20:59, Brad Johnson wrote:
>>>>>>>>>>>> We have a 2 node cluster with a single resource. The resource
>>>>>>>>>>>> must
>>>>>>>>>>>> run
>>>>>>>>>>>> on only a single node at one time. Using the pacemaker:ocf:ping
>>>>>>>>>>>> RA
>>>>>>>>>>>> we
>>>>>>>>>>>> are pinging a WAN gateway and a LAN host on each node so the
>>>>>>>>>>>> resource
>>>>>>>>>>>> runs on the node with the greatest connectivity. The problem is
>>>>>>>>>>>> when
>>>>>>>>>>>> a
>>>>>>>>>>>> ping host goes down (so both nodes lose connectivity to it), the
>>>>>>>>>>>> resource moves to the other node due to timing differences in how
>>>>>>>>>>>> fast
>>>>>>>>>>>> they update the score attribute. The dampening value has no
>>>>>>>>>>>> effect,
>>>>>>>>>>>> since it delays both nodes by the same amount. These unnecessary
>>>>>>>>>>>> fail-overs aren't acceptable since they are disruptive to the
>>>>>>>>>>>> network
>>>>>>>>>>>> for no reason.
>>>>>>>>>>>> Is there a way to dampen the ping update by different amounts on
>>>>>>>>>>>> the
>>>>>>>>>>>> active and passive nodes? Or some other way to configure the
>>>>>>>>>>>> cluster
>>>>>>>>>>>> to
>>>>>>>>>>>> try to keep the resource where it is during these tie score
>>>>>>>>>>>> scenarios?
>>>>>>>>> location pingd-constraint group_1 \
>>>>>>>>>   rule $id="pingd-constraint-rule" pingd: defined pingd
>>>>>>>>>
>>>>>>>>> May I suggest that you simply change this constraint to
>>>>>>>>>
>>>>>>>>> location pingd-constraint group_1 \
>>>>>>>>>   rule $id="pingd-constraint-rule" \
>>>>>>>>>     -inf: not_defined pingd or pingd lte 0
>>>>>>>>>
>>>>>>>>> That way, only a host that definitely has _no_ connectivity carries
>>>>>>>>> a
>>>>>>>>> -INF score for that resource group. And I believe that is what you
>>>>>>>>> really want, rather than take the actual ping score as a placement
>>>>>>>>> weight (your "best connectivity" approach).
>>>>>>>>>
>>>>>>>>> Just my 2 cents, though.
>>>>>>>>>
>>>>>>>> Even though this approach was recommended many times, there is a
>>>>>>>> problem
>>>>>>>> with it.
>>>>>>>> What if all nodes for some reason are not able to ping ?
>>>>>>>> This rule would cause a resource to be brought down completely,
>>>>>>>> whereas
>>>>>>>> if you use "best connectivity" approach it will stay up where it was
>>>>>>>> before
>>>>>>>> network failed.
>>>>>>> If the outside[1] world can't reach the cluster, is there much benefit
>>>>>>> in having it running?
>>>>>>>
>>>>>>> [1] Substitute "outside" for wherever your users are, hopefully you
>>>>>>> picked a ping node from the same area.
>>>>>>>
>>>>>>>> Vadym
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>
>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>> Getting started:
>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>> Bugs:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>
>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>> Getting started:
>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>> Bugs:
>>>>>>>
>>>>>>>
>>>>>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>>>> _______________________________________________
>>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs:
>>>>>>
>>>>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>>>>
>>>>> _______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs:
>>>>>
>>>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs:
>>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs:
>>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs:
>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker

More information about the Pacemaker mailing list