[ClusterLabs] Detect when corosync has started

[Brian Campbell]

I'm running Corosync/Pacemaker on an Ubuntu derivative, and to make
them easier to manage wrote upstart jobs to start them up rather than
using the init scripts.

After doing so, my config scripts, which start up corosync and
pacemaker and then configure them appropriately, start failing, I'm
assuming because the upstart job returns slightly more quickly than
the init scripts did. If I add a short sleep before configuring them,
it succeeds.

However, that short sleep would make every configuration operation a
little bit more slow, so I don't want to do that every time I
reconfigure. I would like instead to check if corosync is functional,
and then poll it a few times until it finishes starting if it's not
running, rather than just unconditionally sleeping for an arbitrary
amount of time.

Is checking the return value from "corosync-cfgtool -s" the best way
to determine this? On systems with corosync running, it returns 0,
while on those where it's not running, it returns 1. Is that a
reliable way to tell if it's up and responding?

Likewise, what's the best way to tell if Pacemaker is up, and able to
be configured? Just check the status of "crm_mon -X"? That does seem
to return success when it's running, and failures when it's not, I
just want to check to see if that will be reliable when it's being
started as well.

-- Brian