[Pacemaker] Frustrating fun with Pacemaker / CentOS / Apache

[Andrew Beekhof]

On Tue, Feb 16, 2010 at 8:47 PM, Paul Graydon <paul at ehawaii.gov> wrote:
> I've been hitting my head against the same brick wall all day today and its
> really beginning to get frustrating.
>
> Overview:   two test servers straight forward CentOS-5.4 servers running
> Apache, and pacemaker on top of OpenAIS.  I've used one of the documented
> setups as the template, with a VIP set up for them to use, and apache set up
> to be monitored.  When it tries to start the httpd process through the CRM
> it hits this really odd error:
>
> Feb 16 09:16:35 test1 lrmd: [19085]: info: RA output:
> (failover-apache:start:stderr) [Tue Feb 16 09:16:35 2010] [warn] worker
> http://ono.:8080/koko already used by another worker
> Feb 16 09:16:35 test1 lrmd: [19085]: info: RA output:
> (failover-apache:start:stderr) (98)Address already in use: make_sock: could
> not bind to address [::]:80
> Feb 16 09:16:35 test1 lrmd: [19085]: info: RA output:
> (failover-apache:start:stderr) (98)Address already in use: make_sock: could
> not bind to address 0.0.0.0:80 no listening sockets available, shutting down
> Unable to open logs
>
> The first error doesn't concern me particularly, it's a known Apache bug
> relating to the proxy module that doesn't actually break anything.  It's the
> binding errors that are bothering me and presumably what is stopping
> pacemaker from starting the service successfully.  Whats really odd about
> that error is I can run "/etc/init.d/httpd start" quite happily myself and
> it works.  There is absolutely nothing sitting listening on port 80 at all
> for it to struggle with.  Occasionally it seems to start it but I've no idea
> why it will succeed then when it fails in the large majority of the time.
>  Really wild stab in the dark, but is there a chance pacemaker is attempting
> to start the httpd process multiple times?

Unlikely, usually its caused by LSB services being told to start at boot time.


> After a while trying to restart the resource group starts throwing up:
> "Error performing operation: Required data for this CIB API call not found"
> with no obvious way to clear that message (nor documentation to that effect
> that I can find?)

Thats not good, can you show us the logs for some context?

> crm(live)configure# show
> node test1 \
>        attributes standby="off"
> node test2 \
>        attributes standby="off"
> primitive failover-apache lsb:httpd \
>        op monitor interval="15s" \
>        meta target-role="Started"
> primitive failover-ip ocf:heartbeat:IPaddr \
>        params ip="10.0.0.10" \
>        op monitor interval="5s" \
>        meta target-role="Started"
> group web-cluster failover-ip failover-apache \
>        meta target-role="Started"
> property $id="cib-bootstrap-options" \
>        stonith-enabled="false" \
>        no-quorum-policy="ignore" \
>        dc-version="1.0.7-d3fa20fc76c7947d6de66db7e52526dc6bd7d782" \
>        cluster-infrastructure="openais" \
>        expected-quorum-votes="2"
>
>
> The other thing I'm struggling to figure out from the documentation is
> whether or not it's possible to choose what order the items get started in?

Ordering constraints:
   http://www.clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-resource-ordering.html

Although, you're using a group, so you'd need to change the order in
which the resources are listed:
   http://www.clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/ch-advanced-resources.html#group-resources

>  Ideally I'd like to bind Apache to the VIP only, not every IP address,
> however pacemaker isn't migrating the IP address until after it tries to
> start apache.  IP address migration happens successfully every single time,
> never a hassle there.
>
> The documentation does seem to make a large number of assumptions about what
> users do or don't know about pacemaker style clustering, and it's been far
> from a simple process to implement what should be a straightforward 2 node
> failover.

Did you try the "cluster- from scratch" doc?

> There is no obvious way to register on the wiki

Its a spam issue unfortunately...

> so I can't even
> offer suggestions or thoughts on the discussion page.

>
> Does anyone have any suggestions as to how to fix the startup problem or
> what might be causing it?
>
> --
> Paul Graydon
> Senior Systems Administrator
> Hawaii Information Consortium
> Internet Portal Partner with the Aloha state
> 808-695-4619 office
> 808-695-4618 fax
> paul@
> *********************************************
> CONFIDENTIALITY NOTICE:
> This email and any attachments are confidential.  If you
> are not the intended recipient, you do not have permission
> to disclose, copy, distribute, or open any attachments.  If
> you have received this email in error, please notify us
> immediately by returning it to the sender and delete this
> copy from your system.
>
> Thank you.
> Hawaii Information Consortium, LLC
> **********************************************
>
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>