[ClusterLabs] Colocation and ordering with live migration

[Pavel Levshin]

Hello.

We are trying to migrate our services to relatively fresh version of 
cluster software. It is RHEL 7 with pacemaker 1.1.13-10. I’ve faced a 
problem when live migration of virtual machines is allowed. In short, I 
need to manage libvirtd, and I cannot set proper ordering constraints to 
stop libvirtd only after all virtual machines are stopped. This leads to 
failing migration and STONITH when I try to shutdown one of the nodes.

If live migration Is disabled, then all works well.

Cluster config follows:

==

Cluster Name: smartbvcluster

Corosync Nodes:

bvnode1 bvnode2

Pacemaker Nodes:

bvnode1 bvnode2

Resources:

Clone: dlm-clone

   Meta Attrs: interleave=true ordered=true

   Resource: dlm (class=ocf provider=pacemaker type=controld)

    Operations: start interval=0s timeout=90 (dlm-start-interval-0s)

                stop interval=0s timeout=100 (dlm-stop-interval-0s)

                monitor interval=30s (dlm-monitor-interval-30s)

Clone: clvmd-clone

   Meta Attrs: interleave=true ordered=true

   Resource: clvmd (class=ocf provider=heartbeat type=clvm)

    Operations: start interval=0s timeout=90 (clvmd-start-interval-0s)

                stop interval=0s timeout=90 (clvmd-stop-interval-0s)

                monitor interval=30s (clvmd-monitor-interval-30s)

Clone: cluster-config-clone

   Meta Attrs: interleave=true

   Resource: cluster-config (class=ocf provider=heartbeat type=Filesystem)

    Attributes: device=/dev/vg_bv_shared/cluster-config 
directory=/opt/cluster-config fstype=gfs2 options=noatime

    Operations: start interval=0s timeout=60 
(cluster-config-start-interval-0s)

                stop interval=0s timeout=60 
(cluster-config-stop-interval-0s)

                monitor interval=10s on-fail=fence OCF_CHECK_LEVEL=20 
(cluster-config-monitor-interval-10s)

Resource: vm_smartbv1 (class=ocf provider=heartbeat type=VirtualDomain)

   Attributes: config=/opt/cluster-config/libvirt/qemu/smartbv1.xml 
hypervisor=qemu:///system migration_transport=tcp

   Meta Attrs: allow-migrate=true

   Operations: start interval=0s timeout=90 (vm_smartbv1-start-interval-0s)

               stop interval=0s timeout=90 (vm_smartbv1-stop-interval-0s)

               monitor interval=10 timeout=30 
(vm_smartbv1-monitor-interval-10)

Resource: vm_smartbv2 (class=ocf provider=heartbeat type=VirtualDomain)

   Attributes: config=/opt/cluster-config/libvirt/qemu/smartbv2.xml 
hypervisor=qemu:///system migration_transport=tcp

   Meta Attrs: target-role=started allow-migrate=true

   Operations: start interval=0s timeout=90 (vm_smartbv2-start-interval-0s)

               stop interval=0s timeout=90 (vm_smartbv2-stop-interval-0s)

               monitor interval=10 timeout=30 
(vm_smartbv2-monitor-interval-10)

Clone: libvirtd-clone

   Meta Attrs: interleave=true

   Resource: libvirtd (class=systemd type=libvirtd)

    Operations: monitor interval=60s (libvirtd-monitor-interval-60s)

Stonith Devices:

Resource: ilo.bvnode2 (class=stonith type=fence_ilo4)

   Attributes: ipaddr=ilo.bvnode2 login=hacluster passwd=s 
pcmk_host_list=bvnode2 privlvl=operator

   Operations: monitor interval=60s (ilo.bvnode2-monitor-interval-60s)

Resource: ilo.bvnode1 (class=stonith type=fence_ilo4)

   Attributes: ipaddr=ilo.bvnode1 login=hacluster passwd=s 
pcmk_host_list=bvnode1 privlvl=operator

   Operations: monitor interval=60s (ilo.bvnode1-monitor-interval-60s)

Fencing Levels:

Node: bvnode1

   Level 10 - ilo.bvnode1

Node: bvnode2

   Level 10 - ilo.bvnode2

Location Constraints:

   Resource: ilo.bvnode1

     Disabled on: bvnode1 (score:-INFINITY) 
(id:location-ilo.bvnode1-bvnode1--INFINITY)

   Resource: ilo.bvnode2

     Disabled on: bvnode2 (score:-INFINITY) 
(id:location-ilo.bvnode2-bvnode2--INFINITY)

Ordering Constraints:

   start dlm-clone then start clvmd-clone (kind:Mandatory) 
(id:order-dlm-clone-clvmd-clone-mandatory)

   start clvmd-clone then start cluster-config-clone (kind:Mandatory) 
(id:order-clvmd-clone-cluster-config-clone-mandatory)

   start cluster-config-clone then start libvirtd-clone (kind:Mandatory) 
(id:order-cluster-config-clone-libvirtd-clone-mandatory)

   start libvirtd-clone then start vm_smartbv2 (kind:Optional) 
(id:order-libvirtd-clone-vm_smartbv2-Optional)

   start libvirtd-clone then start vm_smartbv1 (kind:Optional) 
(id:order-libvirtd-clone-vm_smartbv1-Optional)

Colocation Constraints:

   clvmd-clone with dlm-clone (score:INFINITY) 
(id:colocation-clvmd-clone-dlm-clone-INFINITY)

   cluster-config-clone with clvmd-clone (score:INFINITY) 
(id:colocation-cluster-config-clone-clvmd-clone-INFINITY)

   libvirtd-clone with cluster-config-clone (score:INFINITY) 
(id:colocation-libvirtd-clone-cluster-config-clone-INFINITY)

   vm_smartbv1 with libvirtd-clone (score:INFINITY) 
(id:colocation-vm_smartbv1-libvirtd-clone-INFINITY)

   vm_smartbv2 with libvirtd-clone (score:INFINITY) 
(id:colocation-vm_smartbv2-libvirtd-clone-INFINITY)

Resources Defaults:

No defaults set

Operations Defaults:

No defaults set

Cluster Properties:

cluster-infrastructure: corosync

cluster-name: smartbvcluster

dc-version: 1.1.13-10.el7_2.4-44eb2dd

have-watchdog: false

last-lrm-refresh: 1475752213

maintenance-mode: false

no-quorum-policy: freeze

start-failure-is-fatal: false

stonith-enabled: true

==

Crm_simulate output is attached. What exactly means “dashed-arrows 
indicate dependencies that are not present in the transition graph”? 
What shall I do to make them present?


--

Pavel Levshin


-------------- next part --------------
A non-text attachment was scrubbed...
Name: graph.dot.png
Type: image/png
Size: 244323 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20161010/0f79879c/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graph.xml
Type: text/xml
Size: 14796 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20161010/0f79879c/attachment-0003.xml>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pe-input-2834.bz2
Type: application/octet-stream
Size: 3914 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20161010/0f79879c/attachment-0003.obj>