[ClusterLabs] [Problem:pacemaker_remote] State transition does a loop.

Thu Aug 13 05:50:18 UTC 2015

Hi All,

We confirmed movement of pacemaker_remote.(version:pacemaker-ad1f397a8228a63949f86c96597da5cecc3ed977)

We try complicated cluster constitution.

It is the following cluster constitution.
 * bl460g8n1(Pacemaker host)
 * bl460g8n2(Pacemaker host)
 * bl460g8n3(KVM host - pacemaker_remote moves.)
 * bl460g8n4(KVM host - pacemaker_remote moves.)
 * pgsr01(Guest on the bl460g8n3 host - pacemaker_remote moves.)
 * pgsr02(Guest on the bl460g8n4 host - pacemaker_remote moves.)

Step 1) We send the CLI file.(test-3065-mini.crm)

Step 2) However, the resource of VirtualDomain is not started.
[root at bl460g8n1 ~]# crm_mon -1 -Af
Last updated: Thu Aug 13 14:44:41 2015          Last change: Thu Aug 13 14:40:33 2015 by bl460g8n4 via crm_resource on bl460g8n1
Stack: corosync
Current DC: bl460g8n1 (version 1.1.13-ad1f397) - partition with quorum
6 nodes and 12 resources configured

Online: [ bl460g8n1 bl460g8n2 ]
RemoteOnline: [ bl460g8n3 bl460g8n4 ]
RemoteOFFLINE: [ pgsr01 pgsr02 ]

 bl460g8n3      (ocf::pacemaker:remote):        Started bl460g8n1
 bl460g8n4      (ocf::pacemaker:remote):        Started bl460g8n1
 Resource Group: grpStonith1
     prmStonith1-2      (stonith:external/ipmi):        Started bl460g8n2
 Resource Group: grpStonith2
     prmStonith2-2      (stonith:external/ipmi):        Started bl460g8n1
 Resource Group: grpStonith3
     prmStonith3-2      (stonith:external/ipmi):        Started bl460g8n1
 Resource Group: grpStonith4
     prmStonith4-2      (stonith:external/ipmi):        Started bl460g8n1
 Resource Group: grpStonith5
     prmStonith5-2      (stonith:external/libvirt):     Started bl460g8n1
 Resource Group: grpStonith6
     prmStonith6-2      (stonith:external/libvirt):     Started bl460g8n1
(snip)

State transition seems to do a loop somehow or other.(pe-input-6, pe-input-6.png)
----------------------------
Aug 13 14:40:36 bl460g8n1 crmd[16712]: warning: Transition 6 (Complete=24, Pending=0, Fired=0, Skipped=0, Incomplete=15, Source=/var/lib/pacemaker/pengine/pe-input-6.bz2): Terminated
Aug 13 14:40:36 bl460g8n1 crmd[16712]: warning: Transition failed: terminated
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: Graph 6 with 39 actions: batch-limit=39 jobs, network-delay=0ms
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   21]: Pending rsc op pgsr01_monitor_3000                 on bl460g8n1 (priority: 0, waiting:  20)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   20]: Pending rsc op pgsr01_start_0                      on bl460g8n1 (priority: 0, waiting:  24)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   23]: Pending rsc op pgsr02_monitor_3000                 on bl460g8n1 (priority: 0, waiting:  22)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   22]: Pending rsc op pgsr02_start_0                      on bl460g8n1 (priority: 0, waiting:  26)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   25]: Pending rsc op remoteVM1_monitor_30000             on bl460g8n3 (priority: 0, waiting:  24)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   24]: Pending rsc op remoteVM1_start_0                   on bl460g8n3 (priority: 0, waiting:  4)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   14]: Pending rsc op remoteVM1_monitor_0                 on pgsr02 (priority: 0, waiting:  22)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   11]: Pending rsc op remoteVM1_monitor_0                 on pgsr01 (priority: 0, waiting:  20)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   27]: Pending rsc op remoteVM2_monitor_30000             on bl460g8n4 (priority: 0, waiting:  26)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   26]: Pending rsc op remoteVM2_start_0                   on bl460g8n4 (priority: 0, waiting:  4)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   15]: Pending rsc op remoteVM2_monitor_0                 on pgsr02 (priority: 0, waiting:  22)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   12]: Pending rsc op remoteVM2_monitor_0                 on pgsr01 (priority: 0, waiting:  20)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   31]: Completed pseudo op grpStonith1_running_0          on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   30]: Completed pseudo op grpStonith1_start_0            on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   29]: Completed rsc op prmStonith1-2_monitor_3600000     on bl460g8n2 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   28]: Completed rsc op prmStonith1-2_start_0             on bl460g8n2 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   37]: Completed pseudo op grpStonith2_running_0          on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   36]: Completed pseudo op grpStonith2_start_0            on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   35]: Completed rsc op prmStonith2-2_monitor_3600000     on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   34]: Completed rsc op prmStonith2-2_start_0             on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   43]: Completed pseudo op grpStonith3_running_0          on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   42]: Completed pseudo op grpStonith3_start_0            on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   41]: Completed rsc op prmStonith3-2_monitor_3600000     on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   40]: Completed rsc op prmStonith3-2_start_0             on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   49]: Completed pseudo op grpStonith4_running_0          on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   48]: Completed pseudo op grpStonith4_start_0            on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   47]: Completed rsc op prmStonith4-2_monitor_3600000     on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   46]: Completed rsc op prmStonith4-2_start_0             on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   55]: Completed pseudo op grpStonith5_running_0          on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   54]: Completed pseudo op grpStonith5_start_0            on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   53]: Completed rsc op prmStonith5-2_monitor_3600000     on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   52]: Completed rsc op prmStonith5-2_start_0             on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   61]: Completed pseudo op grpStonith6_running_0          on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   60]: Completed pseudo op grpStonith6_start_0            on N/A (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   59]: Completed rsc op prmStonith6-2_monitor_3600000     on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   58]: Completed rsc op prmStonith6-2_start_0             on bl460g8n1 (priority: 0, waiting: none)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   13]: Pending rsc op probe_complete-pgsr02               on pgsr02 (priority: 1000000, waiting:  14 15)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action   10]: Pending rsc op probe_complete-pgsr01               on pgsr01 (priority: 1000000, waiting:  11 12)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: [Action    4]: Pending pseudo op probe_complete                   on N/A (priority: 0, waiting:  10 13)
Aug 13 14:40:36 bl460g8n1 crmd[16712]: info: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Aug 13 14:40:36 bl460g8n1 crmd[16712]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]

----------------------------

Is the loop of this state transition not Bug?
In addition, by a change of the setting, can we send this CLI file definitely?

 * I registered these contents with Bugzilla.(http://bugs.clusterlabs.org/show_bug.cgi?id=5248)
 * In addition, I attached crm_report to Bugzilla.

Best Regards,
Hideo Yamauchi.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-3065-mini.crm
Type: application/octet-stream
Size: 7659 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20150813/6085fd70/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pe-input-6.dot
Type: application/msword
Size: 6177 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20150813/6085fd70/attachment-0003.dot>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pe-input-6.png
Type: image/png
Size: 233612 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20150813/6085fd70/attachment-0003.png>