[Pacemaker] cib_process_diff: Failed application of an update diff
Johan Huysmans
johan.huysmans at inuits.be
Tue Jul 16 03:37:21 EDT 2013
Hi,
Attached is some more logging of the failed diff:
I hope this is sufficient information to help investigate the problem.
Thx.
Johan
On 15-07-13 03:02, Andrew Beekhof wrote:
> On 11/07/2013, at 5:09 PM, Johan Huysmans <johan.huysmans at inuits.be> wrote:
>
>> Hi,
>>
>> Sorry about the missing info, here it is:
>> OS: CentOS 6.4
>> CoroSync: 1.4.1-15
>> Pacemaker: 1.1.10-1.el6-3463b39 (rc6)
>>
>>
>> Any more info needed to investigate this error?
> Could you set PCMK_trace_functions=apply_xml_diff ( http://blog.clusterlabs.org/blog/2013/pacemaker-logging/ ) and re-test?
> I've been trying to reproduce locally without success.
>
> You'll need to send the cluster _log_ file, not _syslog_.
> Look out for a syslog message like:
>
> pacemakerd[1823]: notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
>
> which tells you where it is.
>
>> On 10-07-13 16:23, Andreas Mock wrote:
>>> Hi Johan,
>>>
>>> I do forget it also often, but more than years ago
>>> you have to give detailed informations about your stack:
>>> - OS
>>> - corosync version (or heartbeat)
>>> - pacemaker version
>>> - agent version
>>> - etc.
>>>
>>> Best regards
>>> Andreas
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Johan Huysmans [mailto:johan.huysmans at inuits.be]
>>> Gesendet: Mittwoch, 10. Juli 2013 15:17
>>> An: The Pacemaker cluster resource manager
>>> Betreff: [Pacemaker] cib_process_diff: Failed application of an update diff
>>>
>>> Hi All,
>>>
>>> Every time a resource fails or recovers or any other action is performed
>>> I see following messages in my log.
>>> Which can be the cause for this problem, how can I see more information
>>> about this message (view the patch / diff which is failing).
>>>
>>>
>>>
>>> stonith-ng[25994]: warning: cib_process_diff: Diff 0.90.29 -> 0.90.30
>>> from local not applied to 0.90.29: Failed application of an update diff
>>> stonith-ng[25994]: notice: update_cib_cache_cb: [cib_diff_notify]
>>> Patch aborted: Application of an update diff failed (-206)
>>>
>>>
>>> thx!
>>> Johan Huysmans
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
info Jul 16 07:28:46 apply_xml_diff(1603):0: Digest mis-match: expected bc4827578f67b6880043d64058311ecf, calculated 58537aaee22cd766026ccc56b1374f96
trace Jul 16 07:28:46 apply_xml_diff(1606):0: 0x97f9680 000000
warning Jul 16 07:28:46 cib_process_diff(676):0: Diff 0.53.27 -> 0.53.28 from local not applied to 0.53.27: Failed application of an update diff
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff crm_feature_set="3.0.7" digest="bc4827578f67b6880043d64058311ecf">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff-removed admin_epoch="0" epoch="53" num_updates="27">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <cib num_updates="27">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <node_state id="DEM-2" crm-debug-origin="do_state_transition">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm_resources>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm_resource id="d_bird">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm_rsc_op last-rc-change="1373958314" id="d_bird_last_failure_0"/>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </lrm_resource>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </lrm_resources>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </lrm>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </node_state>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </cib>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff-removed>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff-added>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <cib epoch="53" num_updates="28" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Tue Jul 16 07:25:39 2013" update-origin="DEM-1" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="DEM-1">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <node_state id="DEM-2" uname="DEM-2" in_ccm="true" crmd="online" join="member" crm-debug-origin="do_update_resource" expected="member">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm_resources>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm_resource id="d_bird" type="bird" class="lsb">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <lrm_rsc_op id="d_bird_last_failure_0" operation_key="d_bird_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="34:1:0:d2bc0959-8658-46dd-b9d1-a7f4f9ac96e8" transition-magic="0:7;34:1:0:d2bc0959-8658-46dd-b9d1-a7f4f9ac96e8" call-id="49" rc-code="7" op-status="0" interval="10000" last-rc-change="1373959726" exec-time="0" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </lrm_resource>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </lrm_resources>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </lrm>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </node_state>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </cib>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff-added>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff>
debug Jul 16 07:28:46 cib_apply_patch_event(863):0: Update didn't apply: Application of an update diff failed
notice Jul 16 07:28:46 update_cib_cache_cb(770):0: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206)
info Jul 16 07:28:46 apply_xml_diff(1603):0: Digest mis-match: expected c632df0e1b9e72e6afd83ceb354c2cc4, calculated cfe73abad8c712547640f36de9bc76cc
trace Jul 16 07:28:46 apply_xml_diff(1606):0: 0x97f9680 000000
warning Jul 16 07:28:46 cib_process_diff(676):0: Diff 0.53.28 -> 0.53.29 from local not applied to 0.53.28: Failed application of an update diff
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff crm_feature_set="3.0.7" digest="c632df0e1b9e72e6afd83ceb354c2cc4">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff-removed admin_epoch="0" epoch="53" num_updates="28">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <cib num_updates="28">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <node_state id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <transient_attributes id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <instance_attributes id="status-DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <nvpair value="1" id="status-DEM-2-fail-count-d_bird"/>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </instance_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </transient_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </node_state>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </cib>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff-removed>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff-added>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <cib epoch="53" num_updates="29" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Tue Jul 16 07:25:39 2013" update-origin="DEM-1" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="DEM-1">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <node_state id="DEM-2" uname="DEM-2" in_ccm="true" crmd="online" join="member" crm-debug-origin="do_update_resource" expected="member">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <transient_attributes id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <instance_attributes id="status-DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <nvpair id="status-DEM-2-fail-count-d_bird" name="fail-count-d_bird" value="2"/>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </instance_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </transient_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </node_state>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </cib>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff-added>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff>
debug Jul 16 07:28:46 cib_apply_patch_event(863):0: Update didn't apply: Application of an update diff failed
notice Jul 16 07:28:46 update_cib_cache_cb(770):0: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206)
info Jul 16 07:28:46 apply_xml_diff(1603):0: Digest mis-match: expected c15ccdf6de832ea7373c1ee471479c04, calculated 8aeb1718e6b13b6dacc5205c9047c3a0
trace Jul 16 07:28:46 apply_xml_diff(1606):0: 0x97f9680 000000
warning Jul 16 07:28:46 cib_process_diff(676):0: Diff 0.53.29 -> 0.53.30 from local not applied to 0.53.29: Failed application of an update diff
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff crm_feature_set="3.0.7" digest="c15ccdf6de832ea7373c1ee471479c04">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff-removed admin_epoch="0" epoch="53" num_updates="29">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <cib num_updates="29">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <node_state id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <transient_attributes id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <instance_attributes id="status-DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <nvpair value="1373958314" id="status-DEM-2-last-failure-d_bird"/>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </instance_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </transient_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </node_state>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </cib>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff-removed>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <diff-added>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <cib epoch="53" num_updates="30" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Tue Jul 16 07:25:39 2013" update-origin="DEM-1" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="1" dc-uuid="DEM-1">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <node_state id="DEM-2" uname="DEM-2" in_ccm="true" crmd="online" join="member" crm-debug-origin="do_update_resource" expected="member">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <transient_attributes id="DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <instance_attributes id="status-DEM-2">
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff <nvpair id="status-DEM-2-last-failure-d_bird" name="last-failure-d_bird" value="1373959726"/>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </instance_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </transient_attributes>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </node_state>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </status>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </cib>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff-added>
trace Jul 16 07:28:46 cib_process_diff(678):0: Discarded diff </diff>
debug Jul 16 07:28:46 cib_apply_patch_event(863):0: Update didn't apply: Application of an update diff failed
notice Jul 16 07:28:46 update_cib_cache_cb(770):0: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206)
More information about the Pacemaker
mailing list