[Pacemaker] Frustrating fun with Pacemaker / CentOS / Apache

[Paul Graydon]

I've been hitting my head against the same brick wall all day today and 
its really beginning to get frustrating.

Overview:   two test servers straight forward CentOS-5.4 servers running 
Apache, and pacemaker on top of OpenAIS.  I've used one of the 
documented setups as the template, with a VIP set up for them to use, 
and apache set up to be monitored.  When it tries to start the httpd 
process through the CRM it hits this really odd error:

Feb 16 09:16:35 test1 lrmd: [19085]: info: RA output: 
(failover-apache:start:stderr) [Tue Feb 16 09:16:35 2010] [warn] worker 
http://ono.:8080/koko already used by another worker
Feb 16 09:16:35 test1 lrmd: [19085]: info: RA output: 
(failover-apache:start:stderr) (98)Address already in use: make_sock: 
could not bind to address [::]:80
Feb 16 09:16:35 test1 lrmd: [19085]: info: RA output: 
(failover-apache:start:stderr) (98)Address already in use: make_sock: 
could not bind to address 0.0.0.0:80 no listening sockets available, 
shutting down Unable to open logs

The first error doesn't concern me particularly, it's a known Apache bug 
relating to the proxy module that doesn't actually break anything.  It's 
the binding errors that are bothering me and presumably what is stopping 
pacemaker from starting the service successfully.  Whats really odd 
about that error is I can run "/etc/init.d/httpd start" quite happily 
myself and it works.  There is absolutely nothing sitting listening on 
port 80 at all for it to struggle with.  Occasionally it seems to start 
it but I've no idea why it will succeed then when it fails in the large 
majority of the time.  Really wild stab in the dark, but is there a 
chance pacemaker is attempting to start the httpd process multiple times?
After a while trying to restart the resource group starts throwing up:
"Error performing operation: Required data for this CIB API call not 
found" with no obvious way to clear that message (nor documentation to 
that effect that I can find?)

crm(live)configure# show
node test1 \
         attributes standby="off"
node test2 \
         attributes standby="off"
primitive failover-apache lsb:httpd \
         op monitor interval="15s" \
         meta target-role="Started"
primitive failover-ip ocf:heartbeat:IPaddr \
         params ip="10.0.0.10" \
         op monitor interval="5s" \
         meta target-role="Started"
group web-cluster failover-ip failover-apache \
         meta target-role="Started"
property $id="cib-bootstrap-options" \
         stonith-enabled="false" \
         no-quorum-policy="ignore" \
         dc-version="1.0.7-d3fa20fc76c7947d6de66db7e52526dc6bd7d782" \
         cluster-infrastructure="openais" \
         expected-quorum-votes="2"


The other thing I'm struggling to figure out from the documentation is 
whether or not it's possible to choose what order the items get started 
in?  Ideally I'd like to bind Apache to the VIP only, not every IP 
address, however pacemaker isn't migrating the IP address until after it 
tries to start apache.  IP address migration happens successfully every 
single time, never a hassle there.

The documentation does seem to make a large number of assumptions about 
what users do or don't know about pacemaker style clustering, and it's 
been far from a simple process to implement what should be a 
straightforward 2 node failover.  There is no obvious way to register on 
the wiki so I can't even offer suggestions or thoughts on the discussion 
page.

Does anyone have any suggestions as to how to fix the startup problem or 
what might be causing it?

-- 
Paul Graydon
Senior Systems Administrator
Hawaii Information Consortium
Internet Portal Partner with the Aloha state
808-695-4619 office
808-695-4618 fax
paul@
*********************************************
CONFIDENTIALITY NOTICE:
This email and any attachments are confidential.  If you
are not the intended recipient, you do not have permission
to disclose, copy, distribute, or open any attachments.  If
you have received this email in error, please notify us
immediately by returning it to the sender and delete this
copy from your system.

Thank you.
Hawaii Information Consortium, LLC
**********************************************