Hi everybody,<br><br>I've fix my problem by inserting followed code to the application consistency checking script.<br>Please let me know whether you found a better solution.<br><br>PS. Thanks for replies<br><br>Vladimir <br>
...<br><font size="1"><span style="font-family: courier new,monospace;">check_marker(){</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # Hardcoded to avoid extra forks.</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> MARKER_COUNT=8</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> NODE_COUNT=4</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> #Operate only on the first online node (from the Online: list)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> if [[ ! "`hostname`" -eq "`crm status |grep Online:| awk '{print $3}'`" ]]; then</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> exit 0;</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> fi</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # Number of started FMS servers</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> ONLINE_FMS=`crm_resource --resource StreamFMS --locate |sed -e "s/\(.*\)://g"|wc -w`</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> #Current state of clone_node-max value</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> CLONE_NODE_MAX=$(crm configure show xml|grep StreamIP-meta_attributes-clone-node-max|sed -e "s/^\(.*\)value=\"//" -e "s/\"\/>//")</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> #How many nodes got a marker.</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> ONLINE_MARKER_COUNT=$(crm status|grep ClusterIP:|grep '(ocf::heartbeat:IPaddr2):'|grep -v Stopped |awk '{print $4}'|sort|uniq |wc -l)</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> REQUIRED_MARKER_COUNT=$(echo -e "a=`echo \"$MARKER_COUNT/$ONLINE_FMS\"| bc`\nb=`echo \"$MARKER_COUNT/$ONLINE_FMS\" | bc -l`\n if (a < b){print ++a} else {print a}" |bc -l )</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> if [ $((ONLINE_MARKER_COUNT*CLONE_NODE_MAX)) -lt "$MARKER_COUNT" ]; then</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # Means: some markers are not allocated!</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # Avoid packet loss:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> crm_resource --resource StreamIP --set-parameter=clone-node-max --meta --parameter-value=$REQUIRED_MARKER_COUNT</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> exit $?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> elif [ "$ONLINE_MARKER_COUNT" -lt $ONLINE_FMS ]; then</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # Means: some nodes do not get workload</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> crm_resource --resource StreamIP --set-parameter=clone-node-max --meta --parameter-value=$REQUIRED_MARKER_COUNT</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> exit $?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> elif [ "$REQUIRED_MARKER_COUNT" -eq "$CLONE_NODE_MAX" ]; then</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # NULL exception</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> if [ $((ONLINE_FMS -1)) -gt '0' ]; then</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> PREDICT_CLONE_NODE_MAX=$(echo -e "a=`echo \"$MARKER_COUNT/($ONLINE_FMS -1)\"| bc`\nb=`echo \"$MARKER_COUNT/($ONLINE_FMS -1)\" | bc -l`\n if (a < b){print ++a} else {print a}" |bc -l )</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> crm_resource --resource StreamIP --set-parameter=clone-node-max --meta --parameter-value=$PREDICT_CLONE_NODE_MAX</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> exit $?</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> fi</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> fi</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">}</span></font><br>...<br><br><br><div class="gmail_quote">
On Fri, Oct 29, 2010 at 4:40 PM, Dan Frincu <span dir="ltr"><<a href="mailto:dfrincu@streamwide.ro">dfrincu@streamwide.ro</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000"><div><div></div><div class="h5">
Hi,<br>
<br>
Vladimir Legeza wrote:
<blockquote type="cite">Hello,<br>
<br>
<div class="gmail_quote">On Fri, Oct 29, 2010 at 12:35 PM, Dan Frincu
<span dir="ltr"><<a href="mailto:dfrincu@streamwide.ro" target="_blank">dfrincu@streamwide.ro</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">Hi,
<div>
<div><br>
<br>
Vladimir Legeza wrote:
<blockquote type="cite"><i>Hello folks.<br>
<br>
I try to setup four ip balanced nodes but, I didn't found the right
way to balance load between nodes when some of them are filed.<br>
<br>
I've done:</i><br>
<br>
[root@node1 ~]# crm configure show<br>
node node1<br>
node node2<br>
node node3<br>
node node4<br>
primitive ClusterIP ocf:heartbeat:IPaddr2 \<br>
params ip="10.138.10.252" cidr_netmask="32"
clusterip_hash="sourceip-sourceport" \<br>
op monitor interval="30s"<br>
clone StreamIP ClusterIP \<br>
meta globally-unique="true" <b>clone-max="8" clone-node-max="2"</b>
target-role="Started" notify="true" ordered="true" interleave="true"<br>
property $id="cib-bootstrap-options" \<br>
dc-version="1.0.9-0a40fd0cb9f2fcedef9d1967115c912314c57438" \<br>
cluster-infrastructure="openais" \<br>
expected-quorum-votes="4" \<br>
no-quorum-policy="ignore" \<br>
stonith-enabled="false"<br>
<br>
<i>When all the nodes are up and running:</i><br>
<br>
[root@node1 ~]# crm status<br>
============<br>
Last updated: Thu Oct 28 17:26:13 2010<br>
Stack: openais<br>
Current DC: node2 - partition with quorum<br>
Version: 1.0.9-0a40fd0cb9f2fcedef9d1967115c912314c57438<br>
4 Nodes configured, 4 expected votes<br>
2 Resources configured.<br>
============<br>
<br>
Online: [ node1 node2 node3 node4 ]<br>
<br>
Clone Set: StreamIP (unique)<br>
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node1<br>
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node1<br>
ClusterIP:2 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:3 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:4 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:5 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:6 (ocf::heartbeat:IPaddr2): Started node4<br>
ClusterIP:7 (ocf::heartbeat:IPaddr2): Started node4<br>
<i><br>
Everything is OK and each node takes 1/4 of all traffic - wonderfull.<br>
But we become to 25% traffic loss if one of them goes down:<br>
</i><br>
</blockquote>
</div>
</div>
Isn't this supposed to be normal behavior in a load balancing
situation, 4 nodes receive 25% of traffic each, one node goes down, the
load balancer notices the failure and directs 33,33% of traffic to the
remaining nodes?<br>
<br>
</div>
</blockquote>
<div> <br>
The only way I see to achive 33...% is to decrease <b>clone-max </b>param
value (that should be multiple of online nodes number)<br>
also <b>clone-max </b>should be changed on the fly (automaticly). <br>
<br>
hmm... Idea is very interesting. =8- )<br>
<b><br>
</b></div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>Just out of curiosity.
<div>
<div><br>
<blockquote type="cite">[root@node1 ~]# crm node standby node1<br>
[root@node1 ~]# crm status<br>
============<br>
Last updated: Thu Oct 28 17:30:01 2010<br>
Stack: openais<br>
Current DC: node2 - partition with quorum<br>
Version: 1.0.9-0a40fd0cb9f2fcedef9d1967115c912314c57438<br>
4 Nodes configured, 4 expected votes<br>
2 Resources configured.<br>
============<br>
<br>
Node node1: standby<br>
Online: [ node2 node3 node4 ]<br>
<br>
Clone Set: StreamIP (unique)<br>
<b> ClusterIP:0 (ocf::heartbeat:IPaddr2): Stopped <br>
ClusterIP:1 (ocf::heartbeat:IPaddr2): Stopped </b><br>
ClusterIP:2 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:3 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:4 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:5 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:6 (ocf::heartbeat:IPaddr2): Started node4<br>
ClusterIP:7 (ocf::heartbeat:IPaddr2): Started node4<br>
<br>
<i>I found the solution (to prevent loosing) by set <b>clone-node-max
</b>to<b> 3</b></i><br>
<br>
[root@node1 ~]# crm resource meta StreamIP set clone-node-max 3<br>
[root@node1 ~]# crm status<br>
============<br>
Last updated: Thu Oct 28 17:35:05 2010<br>
Stack: openais<br>
Current DC: node2 - partition with quorum<br>
Version: 1.0.9-0a40fd0cb9f2fcedef9d1967115c912314c57438<br>
4 Nodes configured, 4 expected votes<br>
2 Resources configured.<br>
============<br>
<br>
<b>Node node1: standby</b><br>
Online: [ node2 node3 node4 ]<br>
<br>
Clone Set: StreamIP (unique)<br>
<b> ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node3</b><br>
ClusterIP:2 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:3 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:4 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:5 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:6 (ocf::heartbeat:IPaddr2): Started node4<br>
ClusterIP:7 (ocf::heartbeat:IPaddr2): Started node4<br>
<br>
<i>The problem is that nothing gonna changed when node1 back
online.</i><br>
<br>
[root@node1 ~]# crm node online node1<br>
[root@node1 ~]# crm status<br>
============<br>
Last updated: Thu Oct 28 17:37:43 2010<br>
Stack: openais<br>
Current DC: node2 - partition with quorum<br>
Version: 1.0.9-0a40fd0cb9f2fcedef9d1967115c912314c57438<br>
4 Nodes configured, 4 expected votes<br>
2 Resources configured.<br>
============<br>
<br>
Online: [ <b>node1</b> node2 node3 node4 ]<br>
<br>
Clone Set: StreamIP (unique)<br>
<b> ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node3</b><br>
ClusterIP:2 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:3 (ocf::heartbeat:IPaddr2): Started node2<br>
ClusterIP:4 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:5 (ocf::heartbeat:IPaddr2): Started node3<br>
ClusterIP:6 (ocf::heartbeat:IPaddr2): Started node4<br>
ClusterIP:7 (ocf::heartbeat:IPaddr2): Started node4<br>
<i><br>
There are NO TRAFFIC on node1.<br>
If I back clone-node-max to 2 - all nodes revert to the original state.</i><br>
<br>
<br>
<br>
So, My question is How to avoid such "hand-made" changes ( or is it
possible to automate<i><b> clone-node-max</b></i> adjustments)? <br>
<br>
Thanks!<br>
</blockquote>
</div>
</div>
You could use location constraints for the clones, something like:<br>
<br>
location StreamIP:0 200: node1<br>
location StreamIP:0 100: node2<br>
<br>
This way if node1 is up, it will run there, but if node1 fails it will
move to node2. And if you don't define resource stickiness, when node1
comes back online, the resource migrates back to it.<br>
</div>
</blockquote>
<div><br>
I already tried to do so, but such configuration is not seems to be
acceptable:<br>
<br>
crm(live)configure# location location_marker_0 StreamIP:0 200: node1<br>
crm(live)configure# commit<br>
element rsc_location: Relax-NG validity error : Expecting an element
rule, got nothing<br>
element rsc_location: Relax-NG validity error : Element constraints has
extra content: rsc_location<br>
element configuration: Relax-NG validity error : Invalid sequence in
interleave<br>
element configuration: Relax-NG validity error : Element configuration
failed to validate content<br>
element cib: Relax-NG validity error : Element cib failed to validate
content<br>
crm_verify[20887]: 2010/10/29_16:00:21 ERROR: main: CIB did not pass <b>DTD/schema
validation</b><br>
Errors found during check: config not valid<br>
<br>
</div>
</div>
</blockquote></div></div>
Here the issue is with the name of the resource in the location
constraint, the name is StreamIP, and it seems it doesn't allow for
referencing child clones, only the parent clone. This is probably the
expected behavior in this case. <br>
<br>
Now you got me thinking, how would such a setup work. The way I see it,
probably there's a better way of doing this.<br>
Create 8 clusterip resources, clusterip{1..8}.<br>
<br>
For each pair of clusterip resource (1+2, 3+4,etc), set a location
constraint of 2x (location clusterip1_on_node1 clusterip1 200: node1,
location clusterip2_on_node1 clusterip2 200: node1) and 6 location
constraints of x for the other nodes.<br>
<br>
This way, you have 2 clusterip resources always preferring one node,
with failover to any of the other 3 available nodes if the current node
fails. Failback is possible when the node comes back online due to the
larger score preference for that node.<br>
<br>
I know this will result in a rather complex set of resources and
constraints, so maybe someone has a better / more simple vision of this.<br>
<br>
Regards,<br><font color="#888888">
<br>
Dan</font><div><div></div><div class="h5"><br>
<blockquote type="cite">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><br>
I haven't tested this, but it should give you a general idea about how
it could be implemented.<br>
<br>
Regards,<br>
<br>
Dan</div>
</blockquote>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
<blockquote type="cite"><br>
<pre><hr size="4" width="90%">
_______________________________________________
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania
</pre>
</div>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
<br>
</blockquote>
</div>
<br>
<pre><hr size="4" width="90%">
_______________________________________________
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a>
</pre>
</blockquote>
<br>
<pre cols="72">--
Dan FRINCU
Systems Engineer
CCNA, RHCE
Streamwide Romania
</pre>
</div></div></div>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
<br></blockquote></div><br>