[Pacemaker] Avoid one node from being a target for resources migration

Tue Jan 13 13:06:18 UTC 2015

> > Then I see that, although node2 clearly knows it's isolated (it doesn't
> see other 2 nodes and does not have quorum)
>
> we don't know that - there are several algorithms for calculating quorum
> and the information isn't included in your output.
> are you using cman, or corosync underneath pacemaker? corosync version?
> pacemaker version? have you set no-quorum-policy?

no-quorum-policy is not set, so, according to
http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-cluster-options.html
, it is "stop - stop all resources in the affected cluster parition". I
suppose this is the right option, but why the resources are not stopped on
the node when this one node of three becomes isolated and the node clearly
sees other nodes as offline (so it knows it's isolated)? What should I
configure in addition?

I'm using corosync+pacemaker, no cman. Below (in quotes) is output of "crm
configure show". Versions are from Ubuntu 14.04, so almost new.

>
> > , it does not stop its services:
> >
> > root at node2:~# crm status
> > Online: [ node2 ]
> > OFFLINE: [ node1 node3 ]
> > Master/Slave Set: ms_drbd [drbd]
> >      Masters: [ node2 ]
> >      Stopped: [ node1 node3 ]
> > Resource Group: server
> >      fs       (ocf::heartbeat:Filesystem):    Started node2
> >      postgresql       (lsb:postgresql):       Started node2
> >      bind9    (lsb:bind9):    Started node2
> >      nginx    (lsb:nginx):    Started node2
> >
> > So is there a way to say pacemaker to shutdown nodes' services when they
> become isolated?
> >
> >
> >
> > On Mon, Jan 12, 2015 at 8:25 PM, David Vossel <dvossel at redhat.com
> <javascript:;>> wrote:
> >
> >
> > ----- Original Message -----
> > > Hello.
> > >
> > > I have 3-node cluster managed by corosync+pacemaker+crm. Node1 and
> Node2 are
> > > DRBD master-slave, also they have a number of other services installed
> > > (postgresql, nginx, ...). Node3 is just a corosync node (for quorum),
> no
> > > DRBD/postgresql/... are installed at it, only corosync+pacemaker.
> > >
> > > But when I add resources to the cluster, a part of them are somehow
> moved to
> > > node3 and since then fail. Note than I have a "colocation" directive to
> > > place these resources to the DRBD master only and "location" with -inf
> for
> > > node3, but this does not help - why? How to make pacemaker not run
> anything
> > > at node3?
> > >
> > > All the resources are added in a single transaction: "cat config.txt |
> crm -w
> > > -f- configure" where config.txt contains directives and "commit"
> statement
> > > at the end.
> > >
> > > Below are "crm status" (error messages) and "crm configure show"
> outputs.
> > >
> > >
> > > root at node3:~# crm status
> > > Current DC: node2 (1017525950) - partition with quorum
> > > 3 Nodes configured
> > > 6 Resources configured
> > > Online: [ node1 node2 node3 ]
> > > Master/Slave Set: ms_drbd [drbd]
> > > Masters: [ node1 ]
> > > Slaves: [ node2 ]
> > > Resource Group: server
> > > fs (ocf::heartbeat:Filesystem): Started node1
> > > postgresql (lsb:postgresql): Started node3 FAILED
> > > bind9 (lsb:bind9): Started node3 FAILED
> > > nginx (lsb:nginx): Started node3 (unmanaged) FAILED
> > > Failed actions:
> > > drbd_monitor_0 (node=node3, call=744, rc=5, status=complete,
> > > last-rc-change=Mon Jan 12 11:16:43 2015, queued=2ms, exec=0ms): not
> > > installed
> > > postgresql_monitor_0 (node=node3, call=753, rc=1, status=complete,
> > > last-rc-change=Mon Jan 12 11:16:43 2015, queued=8ms, exec=0ms): unknown
> > > error
> > > bind9_monitor_0 (node=node3, call=757, rc=1, status=complete,
> > > last-rc-change=Mon Jan 12 11:16:43 2015, queued=11ms, exec=0ms):
> unknown
> > > error
> > > nginx_stop_0 (node=node3, call=767, rc=5, status=complete,
> last-rc-change=Mon
> > > Jan 12 11:16:44 2015, queued=1ms, exec=0ms): not installed
> >
> > Here's what is going on. Even when you say "never run this resource on
> node3"
> > pacemaker is going to probe for the resource regardless on node3 just to
> verify
> > the resource isn't running.
> >
> > The failures you are seeing "monitor_0 failed" indicate that pacemaker
> failed
> > to be able to verify resources are running on node3 because the related
> > packages for the resources are not installed. Given pacemaker's default
> > behavior I'd expect this.
> >
> > You have two options.
> >
> > 1. install the resource related packages on node3 even though you never
> want
> > them to run there. This will allow the resource-agents to verify the
> resource
> > is in fact inactive.
> >
> > 2. If you are using the current master branch of pacemaker, there's a new
> > location constraint option called
> 'resource-discovery=always|never|exclusive'.
> > If you add the 'resource-discovery=never' option to your location
> constraint
> > that attempts to keep resources from node3, you'll avoid having pacemaker
> > perform the 'monitor_0' actions on node3 as well.
> >
> > -- Vossel
> >
> > >
> > > root at node3:~# crm configure show | cat
> > > node $id="1017525950" node2
> > > node $id="13071578" node3
> > > node $id="1760315215" node1
> > > primitive drbd ocf:linbit:drbd \
> > > params drbd_resource="vlv" \
> > > op start interval="0" timeout="240" \
> > > op stop interval="0" timeout="120"
> > > primitive fs ocf:heartbeat:Filesystem \
> > > params device="/dev/drbd0" directory="/var/lib/vlv.drbd/root"
> > > options="noatime,nodiratime" fstype="xfs" \
> > > op start interval="0" timeout="300" \
> > > op stop interval="0" timeout="300"
> > > primitive postgresql lsb:postgresql \
> > > op monitor interval="10" timeout="60" \
> > > op start interval="0" timeout="60" \
> > > op stop interval="0" timeout="60"
> > > primitive bind9 lsb:bind9 \
> > > op monitor interval="10" timeout="60" \
> > > op start interval="0" timeout="60" \
> > > op stop interval="0" timeout="60"
> > > primitive nginx lsb:nginx \
> > > op monitor interval="10" timeout="60" \
> > > op start interval="0" timeout="60" \
> > > op stop interval="0" timeout="60"
> > > group server fs postgresql bind9 nginx
> > > ms ms_drbd drbd meta master-max="1" master-node-max="1" clone-max="2"
> > > clone-node-max="1" notify="true"
> > > location loc_server server rule $id="loc_server-rule" -inf: #uname eq
> node3
> > > colocation col_server inf: server ms_drbd:Master
> > > order ord_server inf: ms_drbd:promote server:start
> > > property $id="cib-bootstrap-options" \
> > > stonith-enabled="false" \
> > > last-lrm-refresh="1421079189" \
> > > maintenance-mode="false"
> > >
> > > _______________________________________________
> > > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org <javascript:;>
> > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org <javascript:;>
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org <javascript:;>
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org <javascript:;>
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20150113/bf0da5ff/attachment.htm>