[Pacemaker] Failing to move around IPaddr2 resource

Fri Feb 24 12:47:47 UTC 2012

On Fri, Feb 24, 2012 at 11:46:07AM +1100, Andrew Beekhof wrote:
> On Mon, Feb 20, 2012 at 12:44 PM, Anlu Wang <anlu at mixpanel.com> wrote:
> > I have three servers that I'm trying to create IP failover on with
> > heartbeat. I have three IPs, one for each machine, and I want an IP to be
> > assigned to a different machine when it goes down. This is all working
> > splendidly.
> >
> > But in addition, I also want an IP to be assigned to a different machine
> > when either the internal OR external network interface goes down. To do
> > this, I have a ping resource on each machine that pings the other 2 machines
> > internal and external ips (so 4 IPs total being pinged on each machine).
> > This is where I'm having problems.
> >
> > When I take down a network interface manually with ifdown, sometimes it
> > fails to stop IP resources on the machines. This is what crm_mon outputs:
> >
> > ============
> > Last updated: Sun Feb 19 19:29:53 2012
> > Stack: Heartbeat
> > Current DC: anlutest2 (32769730-5e5e-40d6-baa0-9748131232da) - partition
> > with quorum
> > Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
> > 3 Nodes configured, unknown expected votes
> > 6 Resources configured.
> > ============
> >
> > Online: [ anlutest1 anlutest3 anlutest2 ]
> >
> > address01       (ocf::heartbeat:IPaddr2):       Started anlutest2
> > (unmanaged) FAILED
> > address02       (ocf::heartbeat:IPaddr2):       Started anlutest3
> > address03       (ocf::heartbeat:IPaddr2):       Started anlutest1
> > (unmanaged) FAILED
> > ping01  (ocf::pacemaker:ping):  Started anlutest1
> > ping02  (ocf::pacemaker:ping):  Started anlutest2
> > ping03  (ocf::pacemaker:ping):  Started anlutest3
> >
> > Failed actions:
> >     address01_stop_0 (node=anlutest2, call=454, rc=1, status=complete):
> > unknown error
> >     address03_stop_0 (node=anlutest1, call=104, rc=1, status=complete):
> > unknown error
> >
> > The reason for this seems to be detailed in the syslog:
> >
> > Feb 19 19:25:06 anlutest1 lrmd: [27108]: info: rsc:address03:104: stop
> > Feb 19 19:25:06 anlutest1 crmd: [27111]: info: process_lrm_event: LRM
> > operation address01_monitor_5000 (call=100, status=1, cib-update=0,
> > confirmed=true) Cancelled
> > Feb 19 19:25:06 anlutest1 crmd: [27111]: info: process_lrm_event: LRM
> > operation address03_monitor_5000 (call=102, status=1, cib-update=0,
> > confirmed=true) Cancelled
> > Feb 19 19:25:06 anlutest1 IPaddr2[32290]: [32350]: INFO: IP status = ok,
> > IP_CIP=
> > Feb 19 19:25:06 anlutest1 IPaddr2[32291]: [32351]: INFO: IP status = ok,
> > IP_CIP=
> > Feb 19 19:25:06 anlutest1 IPaddr2[32290]: [32354]: INFO: ip -f inet addr
> > delete 50.97.234.170/29 dev eth1
> > Feb 19 19:25:06 anlutest1 IPaddr2[32291]: [32355]: INFO: ip -f inet addr
> > delete 50.97.234.172/29 dev eth1
> > Feb 19 19:25:06 anlutest1 crmd: [27111]: info: process_lrm_event: LRM
> > operation address01_stop_0 (call=103, rc=0, cib-update=135, confirmed=true)
> > ok
> > Feb 19 19:25:06 anlutest1 lrmd: [27108]: info: RA output:
> > (address03:stop:stderr) RTNETLINK answers: Cannot assign requested address
> > Feb 19 19:25:06 anlutest1 crmd: [27111]: info: process_lrm_event: LRM
> > operation address03_stop_0 (call=104, rc=1, cib-update=136, confirmed=true)
> > unknown error
> > Feb 19 19:25:07 anlutest1 attrd: [27110]: info: attrd_ha_callback: flush
> > message from anlutest2
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_ha_callback: Update
> > relayed from anlutest2
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_trigger_update:
> > Sending flush op to all hosts for: fail-count-address03 (INFINITY)
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_perform_update: Sent
> > update 377: fail-count-address03=INFINITY
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_ha_callback: Update
> > relayed from anlutest2
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_trigger_update:
> > Sending flush op to all hosts for: last-failure-address03 (1329701107)
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_perform_update: Sent
> > update 379: last-failure-address03=1329701107
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_ha_callback: flush
> > message from anlutest2
> > Feb 19 19:25:08 anlutest1 attrd: [27110]: info: attrd_ha_callback: flush
> > message from anlutest2
> >
> > But I have no idea what the RTNETLINK error is. Googling around seems to
> > show some issues about Ubuntu wireless drivers, but these interfaces are all
> > wired. Does anyone have any idea what is going on? I suspect there might be
> > some sort of weird IP assigning going on, due to the pingd resource not
> > reporting their scores all at the same time maybe?
> 
> Shouldn't be.
> The question is, why would we be /assigning/ an IP during a /stop/ action.

We don't. That's just a stupid error message text...
ip a del dev lo 127.0.0.77/32
RTNETLINK answers: Cannot assign requested address

 ;-)

I assume that you do not have any "primary" address on that nic,
and add several IPaddr2 addresses. The first to start will become the
primary address, the others become secondary addresses (in terms of
linux network configuration).

If you remove the primary address,
all secondary addressees are removed as well.

If you get concurrent stop of several such IPs, those that expect
the IP to be still there may fail. Basically that's some racecondition.

If that is indeed your issue,
you can either assign one static IP on that nic, or
 sysctl -w net.ipv4.conf.all.promote_secondaries=1
(or per device).

> > When I manually go and cleanup the failed nodes, they get properly assigned
> > to the nodes that aren't down, so if we can't resolve the underlying issue,
> > is there a way to automatically attempt to cleanup failed resources a
> > limited number of times?
> 
> I don't think you want to start the IP somewhere else if its still
> active on the original node.
> 
> >
> > My configuration is here, in case there's anything wrong with it.
> 
> Looks like you forgot to attach it.
> 
> >
> > Anlu
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.