[Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?

[Andrew Beekhof]

> On 8 Feb 2015, at 7:09 am, Steven Dake (stdake) <stdake at cisco.com> wrote:
> 
> Hi,
> 
> I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla).  One of the key things we want to do over the next few months is add H/A support to our container tech.  David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers.  That idea is sound.

systemctl or pacemaker-remote in stand-alone mode?
I can imagine the latter more so than the former.

> 
> There is another technology called “super-privileged containers”.  Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file.  I’d like corosync to run in a separate container. 

Separate to even pacemaker?
My container-foo is weak... does they allow traditional IPC mechanisms between containers?

>  These containers will communicate using their normal mechanisms in a super-privileged mode.  We will implement this in Kolla.
> 
> Where I am stuck is how does Pacemaker within a container control other containers  in the host os.  One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process.  Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software.
> 
> An example:
> Lets say a rabbitmq container wants to run:
> 
> The user would run
> kolla-mgr deploy messaging
> 
> This would run a small bit of code to launch the docker container set for messaging.
> 
> Could pacemaker run something like
> 
> Kolla-mgr status messaging
> 
> To control the lifecycle of the processes?

If you wrote an RA that did so, then it can do whatever you like.
However the concept seems prone to internal split-brain given the number of entities wanting a piece of the container.

Will the containers always be in the pacemaker config or only added when someone runs 'deploy'?
How will pacemaker know when to start monitoring?
What should pacemaker do if it detects the container as failed?
How can it tell the difference between a failure and 'kolla-mgr kill messaging'?
Could pacemaker be the 3rd party that does the starting?
If so, perhaps look at the Docker agent.
If not, you will need to make sure pacemaker doesn't start health checking the container until the 3rd party has completely started it.


I'm not saying it cant work as described, I just want to understand the proposal better before I go too much further.

> Or would we be better off with some systemd integration with kolla-mgr?

systemd integration is rarely the answer :-)

> 
> Thoughts welcome
> 
> Regards,
> -steve
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org