[Pacemaker] Centos 70->71 update fails with "Application of an update diff failed (rc=-206)"

[Patrick Zwahlen]

Hi,

I'm running a CentOS 7.0 2-nodes cluster providing iSCSI/SAN features. In order to upgrade to CentOS 7.1, I'm testing the whole process in VMs and it fails. I've now stripped my config down to a pair of DRBD MS with IPADDR2 (cluster.cfg) attached.

>From a running cluster, here are the steps (I'm upgrading node san2):
- put node san2 in standby
- stop/disable pacemaker on san2
- stop/disable corosync on san2
- update san2 to CentOS 7.1 (pacemaker 1.1.10-32.el7_0.1 -> 1.1.12-22.el7_1.1)
- reboot san2
- enable/start corosync on san2. It looks good, rings are fine in "corosync-cfgtool-s")
- enable/start pacemaker on san2

I can see the following in the logs:

/var/log/messages (attached, line #57)
=================
Apr 24 16:18:26 san2 crmd[11759]: notice: erase_xpath_callback: Deletion of "//node_state[@uname='san2.local']/transient_attributes": Application of an update diff failed (rc=-206)

/var/log/pacemaker.log (attached, starting from line #292)
======================
Apr 24 16:18:26 [11754] san2.local        cib:     info: xml_apply_patchset: 	v1 digest mis-match: expected 428c0eb4cd80a4c1ee19b627f6876abd, calculated ffb5456991bd4ed9e5a7774f49e8259d
Apr 24 16:18:26 [11754] san2.local        cib:     info: __xml_diff_object: 	Moved node_state at id (0 -> 6)
Apr 24 16:18:26 [11754] san2.local        cib:     info: __xml_diff_object: 	Moved node_state at uname (1 -> 0)
Apr 24 16:18:26 [11759] san2.local       crmd:   notice: erase_xpath_callback: 	Deletion of "//node_state[@uname='san2.local']/transient_attributes": Application of an update diff failed (rc=-206)
Apr 24 16:18:26 [11754] san2.local        cib:     info: send_sync_request: 	Requesting re-sync from peer
Apr 24 16:18:26 [11754] san2.local        cib:   notice: cib_server_process_diff: 	Not applying diff 0.0.14 -> 0.46.15 (sync in progress)
Apr 24 16:18:26 [11754] san2.local        cib:   notice: cib_server_process_diff: 	Not applying diff 0.0.15 -> 0.46.16 (sync in progress)
Apr 24 16:18:26 [11754] san2.local        cib:   notice: cib_server_process_diff: 	Not applying diff 0.0.16 -> 0.46.17 (sync in progress)

Google doesn't help me in figuring out what might be wrong.

Config was generated with crmsh-2.1-1.4 is that can have an impact.

Any hint would be highly appreciated.

Cheers, Patrick

NOTE: I have kernel modules (scst/zfs) that require reboots when upgrading, so I cannot upgrade both nodes while in unmanaged state. I really need to upgrade one node after the other.


**************************************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager. "postmaster at navixia.com"      Navixia SA
**************************************************************************************
-------------- next part --------------
A non-text attachment was scrubbed...
Name: messages
Type: application/octet-stream
Size: 8963 bytes
Desc: messages
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20150424/172207b3/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cluster.cfg
Type: application/octet-stream
Size: 1998 bytes
Desc: cluster.cfg
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20150424/172207b3/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pacemaker.log
Type: application/octet-stream
Size: 67572 bytes
Desc: pacemaker.log
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20150424/172207b3/attachment-0011.obj>