[Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?

[David Vossel]

----- Original Message -----
> Hi,

Hey Steve, Good to see you around :)

> I am working on Containerizing OpenStack in the Kolla project (
> http://launchpad.net/kolla ). One of the key things we want to do over the
> next few months is add H/A support to our container tech. David Vossel had
> suggested using systemctl to monitor the containers themselves by running
> healthchecking scripts within the containers. That idea is sound.

Knowing what I know about OpenStack HA now, that is a bad choice.

> 
> There is another technology called “super-privileged containers”. Essentially
> it allows more host access for the container, allowing the treatment of

Yep, this is the way to do it. My plan is to have pacemaker running in a container,
and have pacemaker capable of launching resources within containers.

We already have a Docker resource agent. You can find it here,
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker

Using that agent, pacemaker can launch a docker container, and then monitor
the container by performing health checks within the container. Here's an
example of how I'm using this technique to manage a containerized apache
instance.
https://github.com/davidvossel/phd/blob/master/scenarios/docker-apache-ap.scenario#L96


> Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to
> run in a separate container. These containers will communicate using their


I actually already got pacemaker+corosync running in a container for testing
purposes. If you're interested you can checkout some of that work here,
https://github.com/davidvossel/phd/tree/master/lib/docker . The phd_docker_utils.sh
file holds most of the interesting parts. 

> normal mechanisms in a super-privileged mode. We will implement this in
> Kolla.
> 
> Where I am stuck is how does Pacemaker within a container control other
> containers in the host os. One way I have considered is using the docker
> —pid=host flag, allowing pacemaker to communicate directly with the host
> systemctl process. Where I am stuck is our containers don’t run via
> systemctl, but instead via shell scripts that are executed by third party
> deployment software.
> 
> An example:
> Lets say a rabbitmq container wants to run:
> 
> The user would run
> kolla-mgr deploy messaging

yes, and from there kolla-mgr hands the containers off to pacemaker to manage.

kolla is the orchestration, pacemaker is the scheduler for performing those tasks.

> This would run a small bit of code to launch the docker container set for
> messaging.
> 
> Could pacemaker run something like
> 
> Kolla-mgr status messaging
> 
> To control the lifecycle of the processes?
> 
> Or would we be better off with some systemd integration with kolla-mgr?
> 
> Thoughts welcome
> 
> Regards,
> -steve
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>