[Pacemaker] Occasional nonsensical resource agent errors, redux

[Ken Gaillot]

On 11/03/2014 09:26 AM, Dejan Muhamedagic wrote:
> On Mon, Nov 03, 2014 at 08:46:00AM +0300, Andrei Borzenkov wrote:
>> В Mon, 3 Nov 2014 13:32:45 +1100
>> Andrew Beekhof <andrew at beekhof.net> пишет:
>>
>>>
>>>> On 1 Nov 2014, at 11:03 pm, Patrick Kane <pmk at wawd.com> wrote:
>>>>
>>>> Hi all:
>>>>
>>>> In July, list member Ken Gaillot reported occasional nonsensical resource agent errors using Pacemaker (http://oss.clusterlabs.org/pipermail/pacemaker/2014-July/022231.html).
>>>>
>>>> We're seeing similar issues with our install.  We have a 2 node corosync/pacemaker failover configuration that is using the ocf:heartbeat:IPaddr2 resource agent extensively.  About once a week, we'll get an error like this, out of the blue:
>>>>
>>>>    Nov  1 05:23:57 lb02 IPaddr2(anon_ip)[32312]: ERROR: Setup problem: couldn't find command: ip
>>>>
>>>> It goes without saying that the ip command hasn't gone anywhere and all the paths are configured correctly.
>>>>
>>>> We're currently running 1.1.10-14.el6_5.3-368c726 under CentOS 6 x86_64 inside of a xen container.
>>>>
>>>> Any thoughts from folks on what might be happening or how we can get additional debug information to help figure out what's triggering this?
>>>
>>> its pretty much in the hands of the agent.
>>
>> Actually the message seems to be output by check_binary() function
>> which is part of framework.
>
> Someone complained in the IRC about this issue (another resource
> agent though, I think Xen) and they said that which(1) was not
> able to find the program. I'd suggest to do strace (or ltrace)
> of which(1) at that point (it's in ocf-shellfuncs).
>
> The which(1) utility is a simple tool: it splits the PATH
> environment variable and stats the program name appended to each
> of the paths. PATH somehow corrupted or filesystem misbehaving?
> My guess is that it's the former.
>
> BTW, was there an upgrade of some kind before this started
> happening?

I was hoping to have something useful before posting another update, but 
since it's come up again, here's what we've found so far:

* The most common manifestation is the "couldn't find command" error. In 
various instances it "couldn't find" xm, ip or awk. However, we've seen 
two other variations:

   lrmd: [3363]: info: RA output: (pan:monitor:stderr) en-destroy: bad 
variable name

and

   lrmd: [2145]: info: RA output: (ldap-ip:monitor:stderr) 
/usr/lib/ocf/resource.d//heartbeat/IPaddr2: 1: 
/usr/lib/ocf/resource.d//heartbeat/IPaddr2: : Permission denied

The RA in the first case does not use the string "en-destroy" at all; it 
does call a command "xen-destroy". That, to me, is a strong suggestion 
of memory corruption somewhere, whether in the RA, the shell, lrmd or a 
library used by one of those.

* I have not found any bugs in the RA or its included files.

* I tried setting "debug: on" in corosync.conf, but that did not give 
any additional useful information. The resource agent error is still the 
first unusual message in the sequence. Here is an example, giving one 
successful monitor run and then an occurrence of the issue (the nodes 
are a pair of Xen dom0s including pisces, running two Xen domU resources 
pan and nemesis):

Sep 13 20:16:56 pisces lrmd: [3509]: debug: rsc:pan monitor[21] (pid 372)
Sep 13 20:16:56 pisces lrmd: [372]: debug: perform_ra_op: resetting 
scheduler class to SCHED_OTHER
Sep 13 20:16:56 pisces lrmd: [3509]: debug: rsc:nemesis monitor[32] (pid 
409)
Sep 13 20:16:56 pisces lrmd: [409]: debug: perform_ra_op: resetting 
scheduler class to SCHED_OTHER
Sep 13 20:16:56 pisces lrmd: [3509]: info: operation monitor[21] on pan 
for client 3512: pid 372 exited with return code 0
Sep 13 20:16:57 pisces lrmd: [3509]: info: operation monitor[32] on 
nemesis for client 3512: pid 409 exited with return code 0
Sep 13 20:17:06 pisces lrmd: [3509]: debug: rsc:pan monitor[21] (pid 455)
Sep 13 20:17:06 pisces lrmd: [455]: debug: perform_ra_op: resetting 
scheduler class to SCHED_OTHER
Sep 13 20:17:07 pisces lrmd: [3509]: info: RA output: 
(pan:monitor:stderr) /usr/lib/ocf/resource.d//heartbeat/Xen: 71: local:
Sep 13 20:17:07 pisces lrmd: [3509]: info: RA output: 
(pan:monitor:stderr) en-destroy: bad variable name
Sep 13 20:17:07 pisces lrmd: [3509]: info: RA output: (pan:monitor:stderr)
Sep 13 20:17:07 pisces lrmd: [3509]: info: operation monitor[21] on pan 
for client 3512: pid 455 exited with return code 2

* I tried reverting several security updates applied in the month or so 
before we first saw the issue. Reverting the Debian kernel packages to 
3.2.57-3 and then 3.2.54-2 did not help, nor did reverting libxml2 to 
libxml2 2.8.0+dfsg1-7+nmu2. None of the other updates from that time 
look like they could have any effect.

* Regarding libxml2, I did find that Debian had backported an upstream 
patch into its 2.8.0+dfsg1-7+nmu3 that introduced a memory corruption 
bug, which upstream later corrected (the bug never made it into an 
upstream release, but Debian had backported a specific changeset). I 
submitted that as Debian Bug #765770 which was just fixed last week. I 
haven't had a chance to apply that to the affected servers yet, but as 
mentioned above, reverting to the libxml2 before the introduced bug did 
not fix the issue.

* I have not found a way to intentionally reproduce the issue. :-( We 
have had 10 occurrences across 3 two-node clusters in five months. Some 
of the nodes have had only one occurrence during that time, but one pair 
gets the most of them. With the time between occurrences, it's hard to 
do something like strace on lrmd, though that's probably a good way 
forward, scripting something to deal with the output reasonably.

* There does not seem to be any correlation with how long the node has 
been up. Checking RAM usage of corosync and lrmd on all nodes over about 
two weeks shows little to no change, so I don't suspect a leak. Most of 
our errors have occurred in the Xen RA, but probably only because that's 
the RA we use most; we've also seen it in IPaddr2.

* My next idea would be to compile/install the latest versions of at 
least pacemaker and the resource agents. However I am in the middle of 
changing jobs, and unfortunately do not have much time left for this. My 
new job will have plenty of time to spend on pacemaker ;-) so I may be 
able to give updates later. Debian's "jessie" release freezes this week, 
so I'm hoping that I will have time to at least get a test cluster up 
running the somewhat newer versions in that (pacemaker 1.1.10, corosync 
1.4.6).

-- Ken Gaillot <kjgaillo at gleim.com>
Network Operations Center, Gleim Publications