[Pacemaker] A couple of queries regarding the behaviour of ocf:heartbeat:ManageVE

[Dejan Muhamedagic]

Hi,

On Sat, Mar 09, 2013 at 05:29:11PM +0000, Tim Small wrote:
> Hi,
> 
> It looks to me like ocf:heartbeat:ManageVE might do the wrong thing in a
> couple of places, so I'd thought I'd check.
> 
> The resource agent manages openvz containers (i.e. lightweight virtual
> machines AKA "VEs", think chroot++).
> 
> 
> 
> The principle potential problem is the stop operation:
> 
> The metadata for the resource says:
> 
>     <action name="stop" timeout="75" />
> 
> ... however the stop action does this:
> 
> $VZCTL stop $VEID >& /dev/null
>   retcode=$?
> 
>   if [[ $retcode != 0 ]]; then
>     ocf_log err "vzctl stop $VEID returned: $retcode"
>     return $OCF_ERR_GENERIC
>   fi
> 
>   return $OCF_SUCCESS
> 
> 
> 
> When the "vzctl stop" operation is stopped, effectively a "shutdown -h
> now" command gets run within the container, and the container's init
> process attempt to shut down the virtual machines.  If this hasn't
> happened within a certain amount of time (currently hard-coded within
> vzctl to be 120 seconds), e.g. because a service has hung during
> shutdown, or the system is under high load, then the vzctl command gets
> more aggressive and forcibly terminates all of the processes within the
> container.  Whether the shutdown is "normal" or "forced", the vzctl
> return code is still 0 once the container has stopped.  The upshot of
> this is that on an unloaded system (circa 2GHz Intel core2), the vzctl
> command has a max execution time of approx 122 seconds.
> 
> Given the warning at the bottom of:
> 
> http://www.linux-ha.org/doc/dev-guides/_literal_stop_literal_action.html
> 
> It seems to me like <action name="stop" timeout="75" />  (since it
> effectively acts as the default stop timeout in pacemaker), is a bit
> reckless given the behaviour of "vzctl stop" and it should probably be
> 120s + some (e.g. 150s).  BTW as far as I can see vzctl stop has behaved
> like this for a while.
> 
> A more flexible solution might be to make the timeout configurable, but
> in the absence of this, then I think upping the stop action timeout
> seems like the right thing to do.

The value you found is just an advice to the user. You can
define timeout for any operation on a per-resource basis (.e.g.
op stop timeout=4m).

> The second potential problem (the correct behaviour here is a bit less
> clear to me) is with the 'start' command.  It currently starts
> containers asynchronously (i.e. using "vzctl start"  instead of "vzctl
> start --wait"), and then returns immediately.  The monitor operation
> then immediately starts declaring the container to be started, i.e
> return OCF_SUCCESS whilst it still starting up, as well as once it has
> fully started.
> 
> This always-async behaviour effectively defeats any attempt to use

It doesn't really. All resource operations are run in background
anyway (by lrmd).

> something like batch-limit to throttle simultaneous startup of nodes
> (which can obviously be pretty heavy-weight operations - e.g. especially
> when the VEs are relatively fat (for instance, simultaneously starting
> up all the VEs on one the nodes which I have in mind will make the load
> average hit > 100, and could result in the node being fenced for being
> unresponsive - this would then make the same thing happen on the other
> node - splat).
> 
> I'm thinking that ocf:heartbeat:ManageVE should probably default to the
> "vzctl start --wait" case.  I'm not sure if the timeout for start should
> be raised tho' (maybe it should even be lowered as
> http://www.linux-ha.org/doc/dev-guides/_metadata.html states "This is a
> hint to the user what minimal timeout should be configured for the
> action.").

Yes, that's also what I wanted to say a bit earlier.

The start operation should anyway wait until the resource is
completely started, so it should do 'start --wait.'

Thanks,

Dejan

> Any feedback welcome!
> 
> Tim.
> 
> -- 
> South East Open Source Solutions Limited
> Registered in England and Wales with company number 06134732.  
> Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
> VAT number: 900 6633 53  http://seoss.co.uk/ +44-(0)1273-808309
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org