[Pacemaker] Issue in deleting multi state resource

Wed May 7 14:38:09 UTC 2014

I created a multi state resource ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2
(vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2).

Here is the configuration:
==========================
[root at vsanqa11 ~]# pcs config
Cluster Name: vsanqa11_12
Corosync Nodes:

Pacemaker Nodes:
 vsanqa11 vsanqa12

Resources:
 Master: ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2
  Meta Attrs: clone-max=2 globally-unique=false target-role=started
  Resource: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (class=ocf
provider=heartbeat type=vgc-cm-agent.ocf)
   Attributes: cluster_uuid=2be6c088-a1fa-464a-b00d-f4bccb4f5af2
   Operations: monitor interval=30s role=Master timeout=100s
(vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-monitor-interval-30s)
               monitor interval=31s role=Slave timeout=100s
(vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-monitor-interval-31s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2
    Enabled on: vsanqa11 (score:INFINITY)
(id:location-ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY)
    Enabled on: vsanqa12 (score:INFINITY)
(id:location-ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY)
  Resource: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2
    Enabled on: vsanqa11 (score:INFINITY)
(id:location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY)
    Enabled on: vsanqa12 (score:INFINITY)
(id:location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY)
Ordering Constraints:
Colocation Constraints:

Cluster Properties:
 cluster-infrastructure: cman
 dc-version: 1.1.10-14.el6_5.2-368c726
 last-lrm-refresh: 1399466204
 no-quorum-policy: ignore
 stonith-enabled: false

==============================================
When i try to create and delete this resource in a loop, after few
iteration, delete fails as shown below. This can be reproduced easily. I
make sure to unclone resource before deleting the resource. Unclone happens
successfully

Removing Constraint -
location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY
Removing Constraint -
location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY
Attempting to stop: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2...Error:
Unable to stop: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 before deleting
(re-run with --force to force deletion)
Failed to delete resource with uuid: 2be6c088-a1fa-464a-b00d-f4bccb4f5af2

==============================================

Log file snippet of relevant time
============================================

May  7 07:20:12 vsanqa12 vgc-vha-config: /usr/bin/vgc-vha-config --stop
/dev/vgca0_vha
May  7 07:20:12 vsanqa12 crmd[4319]:   notice: do_state_transition: State
transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
May  7 07:20:12 vsanqa12 kernel: VGC: [0000006711341b03:I] Stopped
vHA/vShare instance /dev/vgca0_vha
May  7 07:20:12 vsanqa12 stonith-ng[4315]:   notice: unpack_config: On loss
of CCM Quorum: Ignore
May  7 07:20:12 vsanqa12 vgc-vha-config: Success
May  7 07:20:13 vsanqa12 stonith-ng[4315]:   notice: unpack_config: On loss
of CCM Quorum: Ignore
May  7 07:20:13 vsanqa12 attrd[4317]:   notice: attrd_trigger_update:
Sending flush op to all hosts for:
master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (<null>)
May  7 07:20:13 vsanqa12 attrd[4317]:   notice: attrd_perform_update: Sent
delete 4404: node=vsanqa12,
attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null),
section=status
May  7 07:20:13 vsanqa12 crmd[4319]:   notice: process_lrm_event: LRM
operation vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2_stop_0 (call=1379, rc=0,
cib-update=1161, confirmed=true) ok
May  7 07:20:13 vsanqa12 attrd[4317]:   notice: attrd_perform_update: Sent
delete 4406: node=vsanqa12,
attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null),
section=status
May  7 07:20:12 vsanqa12 kernel: VGC: [0000006711341b03:I] Stopped
vHA/vShare instance /dev/vgca0_vh088-a1fa-464a-b00d-f4bccb4f5af2=(null)
failed: Application of an updMay  7 07:20:12 vsanqa12 stonith-ng[4315]:
notice: unpack_config: On loss of CCM Quorum: Ignore
May  7 07:20:12 vsanqa12 vgc-vha-config: Success
May  7 07:20:13 vsanqa12 stonith-ng[4315]:   notice: unpack_config: On loss
of CCM Quorum: Ignore
May  7 07:20:13 vsanqa12 attrd[4317]:   notice: attrd_trigger_update:
Sending flush op to all hosts for:
master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (<null>)
May  7 07:20:13 vsanqa12 attrd[4317]:   notice: attrd_perform_update: Sent
delete 4404: node=vsanqa12,
attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null),
section=status
May  7 07:20:13 vsanqa12 crmd[4319]:   notice: process_lrm_event: LRM
operation vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2_stop_0 (call=1379, rc=0,
cib-update=1161, confirmed=true) ok
May  7 07:20:13 vsanqa12 attrd[4317]:   notice: attrd_perform_update: Sent
delete 4406: node=vsanqa12,
attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null),
section=status
May  7 07:20:13 vsanqa12 attrd[4317]:  warning: attrd_cib_callback: Update
4404 for master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2=(null) failed:
Application of an update diff failed
May  7 07:20:13 vsanqa12 cib[4314]:  warning: cib_process_diff: Diff
0.6804.2 -> 0.6804.3 from vsanqa11 not applied to 0.6804.2: Failed
application of an update diff
May  7 07:20:13 vsanqa12 cib[4314]:   notice: cib_server_process_diff: Not
applying diff 0.6804.3 -> 0.6804.4 (sync in progress)

[root at vsanqa12 ~]# pcs status
Cluster name: vsanqa11_12
Last updated: Wed May  7 07:24:29 2014
Last change: Wed May  7 07:20:13 2014 via crm_resource on vsanqa11
Stack: cman
Current DC: vsanqa11 - partition with quorum
Version: 1.1.10-14.el6_5.2-368c726
2 Nodes configured
1 Resources configured

Online: [ vsanqa11 vsanqa12 ]

Full list of resources:

 vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2
(ocf::heartbeat:vgc-cm-agent.ocf):      Stopped
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140507/f077c0bd/attachment-0003.html>