[Pacemaker] High load issues

[Steven Dake]

On Thu, 2010-02-04 at 16:09 +0100, Dominik Klein wrote:
> Hi people,
> 
> I'll take the risk of annoying you, but I really think this should not
> be forgotten.
> 
> If there is high load on a node, the cluster seems to have problems
> recovering from that. I'd expect the cluster to recognize that a node is
> unresponsive, stonith it and start services elsewhere.
> 
> By unresponsive I mean not being able to use the cluster's service, not
> being able to ssh into the node.
> 
> I am not sure whether this is an issue of pacemaker (iiuc, beekhof seems
> to think it is not) or corosync (iiuc, sdake seems to think it is not)
> or maybe a configuration/thinking thing on my side (which might just be).
> 
> Anyway, attached you will find a hb_report which covers the startup of
> the cluster nodes, then what it does when there is high load and no
> memory left. Then I killed the load producing things and almost
> immediately, the cluster cleaned up things.
> 
> I had at least expected that after I saw "FAILED" status in crm_mon,
> that after the configured timeouts for stop (120s max in my case), the
> failover should happen, but it did not.
> 
> What I did to produce load:
> * run several "md5sum $file" on 1gig files
> * run several heavy sql statements on large tables
> * saturate(?) the nic using netcat -l on the busy node and netcat -w fed
> by /dev/urandom on another node
> * start a forkbomb script which does "while (true); do bash $0; done;"
> 
> Used versions:
> corosync 1.2.0
> pacemaker 1.0.7
> 64 bit packages from clusterlabs for opensuse 11.1
> 

The forkbomb triggers an OOM situation.  In Linux, when OOM happens
really all bets are off as to what will occur.  I expect that the system
would work properly without the forkbomb.  Could you try that?

Corosync actually works quite well in OOM situations and usually doesn't
detect this as a failure unless the oom killer blows away the corosync
process.  To corosync, the node is fully operational (because it is
designed to work in an OOM situation).

Detecting memory overcommit and doing something about it may be
something we should do with Corosync.

But generally I believe this test case is invalid.  A system should be
properly sized memory wise to handle the applications that are intended
to run on it.  Really sounds like a deployment issue if the systems
don't contain the appropriate ram to run the applications.

I believe there is a way of setting affinity in the OOM killer but it's
been 4 years since I've worked on the kernel fulltime so I don't know
the details.  One option is to set the affinity to always try to blow
away the corosync process.  Then you would get fencing in this
condition.

Regards
-steve

> If you need more information, want me to try patches, whatever, please
> let me know.
> 
> Regards
> Dominik
> _______________________________________________
> Pacemaker mailing list
> Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker