[Pacemaker] [Linux-HA] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10
Andreas Mock
andreas.mock at web.de
Tue Aug 27 19:15:52 EDT 2013
Hi Andrew,
thank you having still an eye on that issue.
I'll do my best to present the requested reports.
Best regards
Andreas Mock
-----Ursprüngliche Nachricht-----
Von: Andrew Beekhof [mailto:andrew at beekhof.net]
Gesendet: Mittwoch, 28. August 2013 00:12
An: General Linux-HA mailing list
Cc: 'The Pacemaker cluster resource manager'
Betreff: Re: [Pacemaker] [Linux-HA] Probably a regression of the linbit drbd
agent between pacemaker 1.1.8 and 1.1.10
On 27/08/2013, at 2:51 PM, Andreas Mock <Andreas.Mock at web.de> wrote:
> Hi Andrew,
>
> as this is a real showstopper at the moment I invested some other
> hours to be sure (as far as possible) not having made an error.
>
> Some additions:
> 1) I mirrored the whole mini drbd config to another pacemaker cluster.
> Same result: pacemaker 1.1.8 works, pacemaker 1.1.10 not
The version of drbd is the same too?
> 2) When I remove the target role Stopped from the drbd ms resource and
> insert the config snippet related to the drbd device via crm -f <file>
> to a lean running pacemaker config (pacemaker cluster options, stonith
> resources), it seems to work. That means one of the nodes gets
> promoted.
>
> Then after stopping 'crm resource stop ms_drbd_xxx' and starting again
> I see the same promotion error as described.
>
> The drbd resource agent is using /usr/sbin/crm_master.
> Is there a possibility that feedback given through this client tool is
> changing the timing behaviour of pacemaker? Or the way transitions are
> scheduled?
> Any idea that may be related to a change in pacemaker?
# git diff --stat Pacemaker-1.1.8..Pacemaker-1.1.10 | tail -n 1
1610 files changed, 109697 insertions(+), 62940 deletions(-)
Needle, meet haystack.
Particularly since I have no idea what that drbd error means.
If you want me to have a look, you'll need to create a crm_report archive of
"works" and "not works".
Logs aren't enough.
>
> Best regards
> Andreas Mock
>
>
> -----Ursprüngliche Nachricht-----
> Von: Andrew Beekhof [mailto:andrew at beekhof.net]
> Gesendet: Dienstag, 27. August 2013 05:02
> An: General Linux-HA mailing list
> Cc: pacemaker at oss.clusterlabs.org
> Betreff: Re: [Pacemaker] [Linux-HA] Probably a regression of the
> linbit drbd agent between pacemaker 1.1.8 and 1.1.10
>
>
> On 27/08/2013, at 3:31 AM, Andreas Mock <Andreas.Mock at web.de> wrote:
>
>> Hi all,
>>
>> while the linbit drbd resource agent seems to work perfectly on
>> pacemaker 1.1.8 (standard software repository) we have problems with
>> the last release 1.1.10 and also with the newest head 1.1.11.xxx.
>>
>> As using drbd is not so uncommon I really hope to find interested
>> people helping me out. I can provide as much debug information as you
>> want.
>>
>>
>> Environment:
>> RHEL 6.4 clone (Scientific Linux 6.4) cman based cluster.
>> DRBD 8.4.3 compiled from sources.
>> 64bit
>>
>> - A drbd resource configured following the linbit documentation.
>> - Manual start and stop (up/down) and setting primary of drbd
>> resource working smoothly.
>> - 2 nodes dis03-test/dis04-test
>>
>>
>>
>> - Following simple config on pacemaker 1.1.8 configure
>> property no-quorum-policy=stop
>> property stonith-enabled=true
>> rsc_defaults resource-stickiness=2
>> primitive r_stonith-dis03-test stonith:fence_mock \
>> meta resource-stickiness="INFINITY" target-role="Started" \
>> op monitor interval="180" timeout="300" requires="nothing" \
>> op start interval="0" timeout="300" \
>> op stop interval="0" timeout="300" \
>> params vmname=dis03-test pcmk_host_list="dis03-test"
>> primitive r_stonith-dis04-test stonith:fence_mock \
>> meta resource-stickiness="INFINITY" target-role="Started" \
>> op monitor interval="180" timeout="300" requires="nothing" \
>> op start interval="0" timeout="300" \
>> op stop interval="0" timeout="300" \
>> params vmname=dis04-test pcmk_host_list="dis04-test"
>> location r_stonith-dis03_hates_dis03 r_stonith-dis03-test \
>> rule $id="r_stonith-dis03_hates_dis03-test_rule" -inf: #uname
>> eq dis03-test
>> location r_stonith-dis04_hates_dis04 r_stonith-dis04-test \
>> rule $id="r_stonith-dis04_hates_dis04-test_rule" -inf: #uname
>> eq dis04-test
>> primitive r_drbd_postfix ocf:linbit:drbd \
>> params drbd_resource="postfix" drbdconf="/usr/local/etc/drbd.conf"
> \
>> op monitor interval="15s" timeout="60s" role="Master" \
>> op monitor interval="45s" timeout="60s" role="Slave" \
>> op start timeout="240" \
>> op stop timeout="240" \
>> meta target-role="Stopped" migration-threshold="2"
>> ms ms_drbd_postfix r_drbd_postfix \
>> meta master-max="1" master-node-max="1" \
>> clone-max="2" clone-node-max="1" \
>> notify="true" \
>> meta target-role="Stopped"
>> commit
>>
>> - Pacemaker is started from scratch
>> - Config above is applied by crm -f <file> where <file> has the above
>> config snippet.
>>
>> - After that crm_mon shows the following status
>> ----------------------8<-------------------------
>> Last updated: Mon Aug 26 18:42:47 2013 Last change: Mon Aug 26
>> 18:42:42 2013 via cibadmin on dis03-test
>> Stack: cman
>> Current DC: dis03-test - partition with quorum
>> Version: 1.1.10-1.el6-9abe687
>> 2 Nodes configured
>> 4 Resources configured
>>
>>
>> Online: [ dis03-test dis04-test ]
>>
>> Full list of resources:
>>
>> r_stonith-dis03-test (stonith:fence_mock): Started dis04-test
>> r_stonith-dis04-test (stonith:fence_mock): Started dis03-test
>> Master/Slave Set: ms_drbd_postfix [r_drbd_postfix]
>> Stopped: [ dis03-test dis04-test ]
>>
>> Migration summary:
>> * Node dis04-test:
>> * Node dis03-test:
>> ----------------------8<-------------------------
>>
>> cat /proc/drbd
>> version: 8.4.3 (api:1/proto:86-101)
>> GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by
>> root at dis03-test,
>> 2013-07-24 17:19:24
>>
>> on both nodes. The drbd resource was shutdown previously in a clean
>> state, so that any node can be the primary.
>>
>> - Now the weird behaviour when trying to start the drbd with crm
>> resource start ms_drbd_postfix
>>
>>
>> Output of crm_mon -1rf
>> ----------------------8<-------------------------
>> Last updated: Mon Aug 26 18:46:33 2013 Last change: Mon Aug 26
>> 18:46:30 2013 via cibadmin on dis04-test
>> Stack: cman
>> Current DC: dis03-test - partition with quorum
>> Version: 1.1.10-1.el6-9abe687
>> 2 Nodes configured
>> 4 Resources configured
>>
>>
>> Online: [ dis03-test dis04-test ]
>>
>> Full list of resources:
>>
>> r_stonith-dis03-test (stonith:fence_mock): Started dis04-test
>> r_stonith-dis04-test (stonith:fence_mock): Started dis03-test
>> Master/Slave Set: ms_drbd_postfix [r_drbd_postfix]
>> Slaves: [ dis03-test ]
>> Stopped: [ dis04-test ]
>>
>> Migration summary:
>> * Node dis04-test:
>> r_drbd_postfix: migration-threshold=2 fail-count=2 last-failure='Mon
>> Aug
>> 26 18:46:30 2013'
>> * Node dis03-test:
>>
>
> Its hard to imagine how pacemaker could cause drbdadm to fail, short
> of leaving the other side promoted while trying to promote another.
> Perhaps the drbd folks could comment on what the error means.
>
>> Failed actions:
>> r_drbd_postfix_promote_0 (node=dis04-test, call=34, rc=1,
>> status=complete, last-rc-change=Mon Aug 26 18:46:29 2013 ,
>> queued=1212ms, exec=0ms
>> ): unknown error
>> ----------------------8<-------------------------
>>
>> In the log of the drbd agent I can find the following when the
>> promoting request is handled on dis03-test
>>
>> ----------------------8<-------------------------
>> ++ drbdadm -c /usr/local/etc/drbd.conf primary postfix
>> 0: State change failed: (-2) Need access to UpToDate data Command
>> 'drbdsetup primary 0' terminated with exit code 17
>> + cmd_out=
>> + ret=17
>> + '[' 17 '!=' 0 ']'
>> + ocf_log err 'postfix: Called drbdadm -c /usr/local/etc/drbd.conf
>> + primary
>> postfix'
>> + '[' 2 -lt 2 ']'
>> + __OCF_PRIO=err
>> + shift
>> ----------------------8<-------------------------
>>
>> While working without problems on pacemaker 1.1.8 it doesn't work here.
>> The error message let me assume that there is a kind of race
>> condition where pacemaker is firing the promotion too early.
>> Probably it has something to do with applying attributes from the
>> drbd resource agent.
>> But this is just a guess and I really don't know.
>>
>> ONE ADDITIONAL information: As soon as I do a resource cleanup on the
>> "defective" node the master is promoted as expected. That means a:
>> crm resource cleanup r_drbd_postfix dis03-test results in the
>> following:
>>
>> ----------------------8<-------------------------
>> Last updated: Mon Aug 26 19:29:38 2013 Last change: Mon Aug 26
>> 19:29:28 2013 via cibadmin on dis04-test
>> Stack: cman
>> Current DC: dis03-test - partition with quorum
>> Version: 1.1.10-1.el6-9abe687
>> 2 Nodes configured
>> 4 Resources configured
>>
>>
>> Online: [ dis03-test dis04-test ]
>>
>> Full list of resources:
>>
>> r_stonith-dis03-test (stonith:fence_mock): Started dis04-test
>> r_stonith-dis04-test (stonith:fence_mock): Started dis03-test
>> Master/Slave Set: ms_drbd_postfix [r_drbd_postfix]
>> Masters: [ dis03-test ]
>> Slaves: [ dis04-test ]
>>
>> Migration summary:
>> * Node dis03-test:
>> * Node dis04-test:
>> ----------------------8<-------------------------
>>
>>
>>
>> I really hope I can get some attention as pacemaker 1.1.10 is a
>> milestone for Andrew and drbd from linbit is pretty sure a building
>> block of many pacemaker based clusters.
>>
>> Cluster log of DC dis03-test at http://pastebin.com/2S9Y6V3P DRBD
>> agent log at http://pastebin.com/ceYNEAhH
>>
>>
>> So, any help welcome.
>>
>> Best regards
>> Andreas Mock
>>
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA at lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
More information about the Pacemaker
mailing list