[Pacemaker] Frustrating fun with Pacemaker / CentOS / Apache

[Paul Graydon]

On 2/18/2010 2:10 AM, Dejan Muhamedagic wrote:
> Hi,
>
> On Wed, Feb 17, 2010 at 04:59:16PM -1000, Paul Graydon wrote:
>    
>> On 2/16/2010 1:04 PM, Angie T. Muhammad wrote:
>>      
>>>
>>> On Wed, Feb 17, 2010 at 12:21 AM, Paul Graydon<paul at ehawaii.gov
>>> <mailto:paul at ehawaii.gov>>  wrote:
>>>
>>>     On 2/16/2010 10:48 AM, Andrew Beekhof wrote:
>>>        
>>>>     The first error doesn't concern me particularly, it's a known
>>>>     Apache bug
>>>>          
>>>>>     relating to the proxy module that doesn't actually break anything.  It's the
>>>>>     binding errors that are bothering me and presumably what is stopping
>>>>>     pacemaker from starting the service successfully.  Whats really odd about
>>>>>     that error is I can run "/etc/init.d/httpd start" quite happily myself and
>>>>>     it works.  There is absolutely nothing sitting listening on port 80 at all
>>>>>     for it to struggle with.  Occasionally it seems to start it but I've no idea
>>>>>     why it will succeed then when it fails in the large majority of the time.
>>>>>       Really wild stab in the dark, but is there a chance pacemaker is attempting
>>>>>     to start the httpd process multiple times?
>>>>>            
>>>>     Unlikely, usually its caused by LSB services being told to start at boot time.
>>>>          
>>>     That was one of the earliest thoughts I had, sorry I meant to put
>>>     this in my first message:
>>>
>>>     # chkconfig --list httpd
>>>     httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
>>>
>>> I suffered from the same problem as you do. It is always highly
>>> recommended to use OCF modules written specifically for your
>>> service rather than LSB. Aside of your stack (openAIS or
>>> Heartbeat), errors will pop up when stopping httpd if you are
>>> using LSB..
>>>
>>> Example configuration:
>>> # crm configure primitive WebServer ocf:heartbeat:apache params
>>> configfile=/etc/httpd/conf/httpd.conf
>>> statusurl=http://127.0.0.1/server-status op monitor interval=30s
>>>
>>> For the above example to work correctly, FIRST you have to do some
>>> editing in httpd.conf:
>>>
>>> 1- Make your listen directive tied to the localhost (for testing):
>>> Listen 127.0.0.1:80<http://127.0.0.1:80>
>>>
>>> 2- Enable viewing server status for localhost:
>>> <Location /server-status>
>>>     SetHandler server-status
>>>     Order deny,allow
>>>     Deny from all
>>>     Allow from 127.0.0.1
>>> </Location>
>>>
>>> 3- Enable extended status:
>>> ExtendedStatus On
>>>
>>>
>>> Other than this, pacemaker will try to request a page from your
>>> server but never finds it, so it considers the server not
>>> responding. It tries to kill it, and you end up with some zombie
>>> processes.
>>>        
>> I started looking into this today to find out whether it was
>> possible to use another URL for testing.  According to the heartbeat
>> script you can specify the parameter statusurl and as long as it has
>> a body and html tag on the page you test it should work.
>>      
> You can also set your own "testregex" which should match the
> output of "statusurl". Since resource agents release 1.0.2,
> apache can also do more thorough tests (see "crm ra info apache"
> or ocf_heartbeat_apache(7)).
>
>    
>> So I thought I'd give it a try, but it failed.  Initially I assumed
>> it was because I hadn't selected a page with the body and html tag
>> (having not noticed that was a necessity) but even when against a
>> page that has them it still failed.  Trying to execute the command
>> it runs came up with a failure for me too, but it appears to be how
>> all the arguments are presented to wget courtesy of "sh -c".
>>
>> It's looking for a positive return from:
>>
>> sh -c wget -O- -q -L http://whatever.url.youprovided | tr '\012' ' '
>> | grep -Ei "</ *body *>[[:space:]]*</ *html *>"
>>
>> Problem is if you cut it down to just that first section:
>> sh -c wget -O- -q -L http://whatever.url.youprovided
>>
>> it pops back and tells you
>> wget: missing URL
>> Usage: wget [OPTION]... [URL]...
>>
>> Try `wget --help' for more options.
>>
>> If you execute wget without using "sh -c" in front of it it sees the
>> URL and parses it successfully.
>>
>> Surrounding the wget string with ' marks, as in:
>> sh -c 'wget -O- -q -L http://whatever.url.youprovided '
>>
>> I'm trying to figure out what other options are available.  Adding
>> in ' marks on line 406 of the ocf heartbeat apache script breaks it!
>>      
> I really don't think there is a need to change anything there.
> Otherwise, apache would never be able to work. If you think you
> found a problem, you can try to wrap the critical part in
> "set -x/+x" and we'll see what the output looks like.
>
> Thanks,
>
> Dejan
>    
I've looked into this with fresh eyes this morning and managed to track 
down the problem to this addition to the related to meta 
target-role="Started"

Not sure quite where I picked that up from, presumably one of the 
configurations I used as a template?  Without setting it as an attribute 
it works fine, tested and retested with and without that addition.

This works:
primitive failover-apache ocf:heartbeat:apache \
         params configfile="/etc/httpd/conf/httpd.conf" 
httpd="/usr/sbin/httpd" port="80" \
         op monitor interval="5s" timeout="20s" 
statusurl="https://valid.test.url/index.html"

This doesn't:
primitive failover-apache ocf:heartbeat:apache \
         params configfile="/etc/httpd/conf/httpd.conf" 
httpd="/usr/sbin/httpd" port="80" \
         op monitor interval="5s" timeout="20s" 
statusurl="https://valid.test.url/index.html" \
     meta target-role="Started"

My understanding of the meta bits is a little weak, and I can't find an 
explanation as to what target-role is actually trying to do.


What I didn't twig last night was that what is logged by the agent as 
being run isn't what is actually being run, i.e. the wget statement is 
actually bracketed by quotes, so it operates correctly.


-- 
Paul Graydon
Senior Systems Administrator
Hawaii Information Consortium
Internet Portal Partner with the Aloha state
808-695-4619 office
808-695-4618 fax
paul at ehawaii.gov
*********************************************
CONFIDENTIALITY NOTICE:
This email and any attachments are confidential.  If you
are not the intended recipient, you do not have permission
to disclose, copy, distribute, or open any attachments.  If
you have received this email in error, please notify us
immediately by returning it to the sender and delete this
copy from your system.

Thank you.
Hawaii Information Consortium, LLC
**********************************************