[Pacemaker] Ordering clones and primitives

[Lars Kellogg-Stedman]

> You should set the interleave=true meta attribute for the clones.
> Hope that that would help. You need collocations as well.

Dejan,

Thanks for the suggestion.  Can you elaborate a little?  The documentation
for the interleave option says:

  Changes the behavior of ordering constraints (between clones/masters) so
  that instances can start/stop as soon as their peer instance has (rather
  than waiting for every instance of the other clone has). Allowed values:
  true, false

I'm not sure exactly what this means.  In particular, I'm not sure what a
"master" is in this case; the documentation only refers to masters in the
context of multi-state resources.  Maybe we could work out an explicit
example?  Let's start with a sample configuration like this:

  primitive def_route ocf:pacemaker:Stateful
  primitive ip_0 ocf:pacemaker:Stateful
  primitive ip_1 ocf:pacemaker:Stateful
  primitive net_route ocf:pacemaker:Stateful
  clone clone_def_route def_route \
          meta interleave="true"
  clone clone_net_route net_route \
          meta interleave="true"

My goal is that:

- net_route must start before def_route, and there must be either ip_0 or
  ip_1 active on a node before either net_route or def_route can start.
- Both ip_0 and ip_1 can run on the same node.
- Only a single instance of net_route and def_route can run on any given
  node.

Without any constraints and everything started, "crm status" looks like
this:

  Online: [ node1 node0 ]

   ip_0   (ocf::pacemaker:Stateful):      Started node1
   ip_1   (ocf::pacemaker:Stateful):      Started node0
   Clone Set: clone_def_route [def_route]
       Started: [ node1 node0 ]
   Clone Set: clone_net_route [net_route]
       Started: [ node1 node0 ]

If I add what I think is the correct collocation constraint (with the
caveat that I don't really understand the semantics of collocating clones
and primitives):

  colocation ip_0_with_routes \
    inf: ip_0 clone_net_route clone_def_route
  colocation ip_1_with_routes \
    inf: ip_1 clone_net_route clone_def_route

I end up with this:

  Online: [ node1 node0 ]

   ip_0   (ocf::pacemaker:Stateful):      Started node1
   ip_1   (ocf::pacemaker:Stateful):      Started node0

All of the clones are missing!  They still show up in the resource list:

  # crm resource list
   ip_0   (ocf::pacemaker:Stateful) Started
   ip_1   (ocf::pacemaker:Stateful) Started
   Clone Set: clone_def_route [def_route]
       Stopped: [ def_route:0 def_route:1 ]
   Clone Set: clone_net_route [net_route]
       Stopped: [ net_route:0 net_route:1 ]

But trying to start them is  no-op.

Thanks for your help!  If we get this working maybe we can turn it into an
example configuration on the wiki.